Commit Graph

267 Commits

Author SHA1 Message Date
Henrik Grimler ca93111034
Revert "[TEMP] CI: disable uploads"
Migration from fosshost to new vps is more or less complete.

This reverts commit 96c22cf398.
2022-12-10 12:18:35 +01:00
Henrik Grimler 96c22cf398
[TEMP] CI: disable uploads
We are migrating from fosshost to another vps, disable package uploads
until migration is complete.

See termux/termux-packages#13804.
2022-12-10 10:00:01 +01:00
Henrik Grimler c004fd4514
enhance(scripts,packages.yml): sync upload steps
In commit ca2e7bcf73 ("ci(packages): Do not unconditionally publish
repo when no changes are made to other repos") the upload step to
fosshost was updated, but not the one for grimler.se.  Sync the upload
steps so that they are nearly identical again.
2022-10-30 15:17:33 +01:00
Henrik Grimler 5c4841e3a8
fix(scripts,packages.yml): fix duplicate line removal
Need to sort the file before running uniq.  Before this fix
subpackages were printed several times in the
built_${repo}_packages.txt file, causing upload step to upload debs
multiple times for no reason..
2022-10-30 15:14:43 +01:00
dependabot[bot] 505f04f2b0
build(deps): bump docker/login-action from 1 to 2 (#12629)
Bumps [docker/login-action](https://github.com/docker/login-action) from 1 to 2.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-30 00:09:05 +01:00
agnostic-apollo 3b7e145b2a Revert "add(package.yaml): add support to only run upload job"
This reverts commit 5b256ab96d.
2022-10-07 05:45:22 +05:00
agnostic-apollo 5e0b5927c0 fix(packages.yml): fix upload job being skipped if only run upload job toggle is enabled since build job is skipped failing needs condition
https://github.com/actions/runner/issues/491#issuecomment-850884422

Related commit 5b256ab9
2022-10-07 05:33:19 +05:00
agnostic-apollo 5b256ab96d add(package.yaml): add support to only run upload job
If `Run only upload job` toggle is enabled, pass anything as packages value since its required and it will be ignored.
2022-10-07 04:57:17 +05:00
Lucinda May Phipps fdc4ee55cc
packages.yml: run one upload job at a time 2022-10-06 22:24:05 +01:00
agnostic-apollo e9dae011e2 fixed(bootstrap_archives.yml): Do not use natural sort while generating bootstraps.sha256sum 2022-09-17 08:22:55 +05:00
babaric-dev e1b1509c6e fixed(bootstrap_archives.yml): fix invalid file parameter for bootstraps github release upload job broken in 4af58fc0
Related pull request #11960 and #11956
2022-09-17 08:10:16 +05:00
babaric-dev 4af58fc016 chore,ci(bootstrap_archives): Add sha256sum to releases 2022-09-16 23:07:02 +08:00
Henrik Grimler edf4586feb
scripts(workflows/packages): update list of packages to uninstall
When we need to get extra disk space in CI.

Previous build for auto update of clvk failed.
2022-09-15 21:30:30 +02:00
Chongyun Lee 8134416035 ci: switch to 22.04 2022-09-13 18:43:38 +08:00
Chongyun Lee 30ec184fb6
Revert "ci: fix packages list"
This reverts commit 0c92e85834.
2022-09-12 22:54:40 +08:00
Chongyun Lee 7204e671aa
Revert "ci: switch to 22.04"
This reverts commit 8a957b5116.
2022-09-12 22:54:34 +08:00
Chongyun Lee 0c92e85834
ci: fix packages list 2022-09-12 22:24:30 +08:00
Chongyun Lee 8a957b5116 ci: switch to 22.04 2022-09-12 22:15:53 +08:00
Butta a06a89a012 upgpkg(swift): 5.6.2 to 5.6.3 2022-09-03 02:24:55 +05:30
Alex fdead17156
chore,ci: use explicit permissions for GitHub Actions
Quoted from #11735:
This commit adds explicit [permissions
section](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions)
to workflows. This is a security best practice because by default
workflows run with [extended set of
permissions](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token)
(except from `on: pull_request` [from external
forks](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/)).
By specifying any permission explicitly all others are set to none. By
using the principle of least privilege the damage a compromised workflow
can do (because of an
[injection](https://securitylab.github.com/research/github-actions-untrusted-input/)
or compromised third party tool or action) is restricted.\r\nIt is
recommended to have [most strict permissions on the top
level](https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions)
and grant write permissions on [job
level](https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs)
case by case.

Co-Authored-By: Alex <93376818+sashashura@users.noreply.github.com>
2022-08-28 08:59:01 +05:30
Leonid Pliushch 47e6537bb2
ci(docker_image.yml): workaround for sometimes unstable ghcr.io
ghcr.io sometimes drops connection when image is pushed from GitHub
Actions. I never observed that behavior when pushing the image built
locally, so don't know what triggers that issue.

The workaround is to just try pushing the image again.
2022-07-23 11:32:57 +03:00
Leonid Pliushch dc3869ad37
enhance(scripts): use GitHub package registry for Docker image
Should provide better (maybe) download speed when using GitHub Actions.
2022-07-23 00:19:08 +03:00
Henrik Grimler 6cf9f35fac
scripts,ci(*): use termux.dev instead of termux.org
We are now mainly promoting and using termux.dev, so lets switch all
the urls.
2022-07-02 10:41:16 +02:00
Henrik Grimler 595969f581
Revert "fix(workflows/packages.yml): Fix uploads to packages.termux.dev by using a working ip address for aptly"
packages.termux.dev should work fully now, so remove workaround.

This reverts commit a3e68e2b89.
2022-07-02 10:24:37 +02:00
Henrik Grimler f5316db82c
fix,ci(packages.yml): specify port when checking repository
Fixes commit a3e68e2b89 ("fix(workflows/packages.yml): Fix uploads
to packages.termux.dev by using a working ip address for aptly").
2022-06-18 19:45:39 +02:00
Henrik Grimler 93e4eb1a75
Revert "Revert "ci(workflows/packages.yml): use packages.termux.dev for uploads""
I reverted the wrong commit.

This reverts commit d090e21390.
2022-06-18 19:08:27 +02:00
Henrik Grimler d090e21390
Revert "ci(workflows/packages.yml): use packages.termux.dev for uploads"
More proper fix by @agnostic-apollo in a3e68e2b89 and 3b76d37dce.

This reverts commit bc9834dfe9.
2022-06-18 18:42:22 +02:00
agnostic-apollo a3e68e2b89 fix(workflows/packages.yml): Fix uploads to packages.termux.dev by using a working ip address for aptly
Related issue #11007
2022-06-18 17:44:33 +05:00
Jia Yuan Lo b469675248
ci(workflows/packages.yml): free disk space when building clvk 2022-06-18 11:54:05 +02:00
Butta 2056aad712 swift: clear up space on CI for longer aarch64 build, until 5.7 is out in a couple months 2022-06-17 16:39:16 +05:30
Henrik Grimler bc9834dfe9
ci(workflows/packages.yml): use packages.termux.dev for uploads
We are migrating from termux.org to termux.dev for all services. .org
redirects to .dev, so should be backward compatible, but url for aptly
scripts need to be updated as they do not handle returned http code
301/302.
2022-06-12 15:15:07 +02:00
agnostic-apollo aecb8bd19d
Fixed(bootstrap_archives): Fix prevent parallel runs implemented in 332e1550
The workflow is not valid. .github/workflows/bootstrap_archives.yml (Line: 9, Col: 10): Unexpected type of value '', expected type: String.
2022-05-29 05:28:33 +05:00
Yaksh Bariya 332e155068
chore,ci(bootstrap_archives): prevent parallel runs
Can lead to race conditions
2022-05-22 14:56:04 +05:30
dev-bz 7c06b80bdb
dart: Update to 2.17.0 (#10672)
* dart: Update to 2.17.0 and add to CI packages that need more disk space
2022-05-15 14:16:32 +05:30
Yaksh Bariya ca2e7bcf73
ci(packages): Do not unconditionally publish repo when no changes are
made to other repos
2022-05-05 18:27:43 +05:30
Henrik Grimler cc441d5dbf
CI: bootstraps: append +apt-android-7 to bootstrap archives
When they are generated from master branch.  This is done since
termux-app now comes in two flavours, one for android >= 5 and one for
android >= 7.
2022-04-28 22:03:18 +02:00
Yaksh Bariya 52c8fd0a41
ci(bootstrap_archives): use specific version 2022-04-28 19:08:59 +05:30
Yaksh Bariya 5edddd95c0
chore,ci(bootstrap_archives): Use Termux's fork of svenstaro/upload-release-action 2022-04-28 15:03:02 +05:30
Henrik Grimler b4a3d50955
CI: fix regex pattern in free additional disk space step 2022-04-28 09:32:20 +02:00
muku Takeda a98d8d10cc
Add rustc-nightly to run Free additional disk space on host 2022-04-27 10:13:28 +02:00
Yaksh Bariya 029da6a2dd
ci: use v3 of actions/{checkout,download-artifact,upload-artifact} 2022-04-26 19:41:35 +05:30
Yaksh Bariya 4c532c0de5
ci: fix "Free Additional disk space" step
Hardcode so tbat it greps built_termux-main_packages.txt instead of
built_packages.txt
2022-04-21 16:59:53 +05:30
Yaksh Bariya d366fcab14
fixup(ci): Fix duplication of package names in
`built_${repo}_packages.txt`

Caused by 56dbb9cf80
2022-04-21 16:04:16 +05:30
Henrik Grimler 56dbb9cf80
CI: specify distribution in repo.json as well
x11-repo uses distribution x11, and root-repo distribution root.
Store this information in the json file as well, and parse it to set
both REPOSITORY_NAME and REPOSITORY_DISTRIBUTION.

Also remove unnecessary `< repo.json` from jq command, jq accepts the
file as an argument.

This fixes upload issues to x11-repo and root-repo.
2022-04-20 16:37:36 +02:00
Yaksh Bariya 04e7833ede
fixup(ci): do not process '%ci:no-build' on 'workflow_dispatch' 2022-04-19 20:55:54 +05:30
Yaksh Bariya e7431f5aea
fixup(ci): do not lint all packages when `%ci:no-build` is added to HEAD commit 2022-04-19 20:51:26 +05:30
Yaksh Bariya d41ad97af5
ci: fix uploads 2022-04-19 07:19:39 +05:30
Yaksh Bariya 98fff5b520
Fix generation of built_$repo_packages.txt 2022-04-19 07:19:28 +05:30
Yaksh Bariya 7669ea0e3b
ci: fix variables 2022-04-18 15:29:38 +05:30
Yaksh Bariya 1b8ab728e0
ci: fix generate build artifact step 2022-04-18 15:08:31 +05:30
Yaksh Bariya 681d108453
ci: do not use local keyword for declaring variables 2022-04-18 14:57:35 +05:30
Yaksh Bariya f0ea9b922b
monorepo: apply changes suggested by buttaface 2022-04-18 13:57:19 +05:30
Yaksh Bariya 52c383b4d1
chore: switch to monorepo 2022-04-18 13:57:13 +05:30
Yaksh Bariya dcaf44f95b
ci: sleep only when aptly_add_to_repo returns with status code 000
It is just a waste of time to sleep for 3 minutes when the packages are
successfully added and aptly returns 200 http code
2022-04-16 14:07:17 +05:30
Yaksh Bariya cac3799e1d
ci: further split jobs to improve log browsing experience 2022-04-16 14:07:12 +05:30
Yaksh Bariya c4ab7bcfd0
attempt2: Fix repository upload problems
Earlier in 5b8b15e7aa, I had configured
our upload scripts to treat failure of `aptly_add_to_repo` as failure to publish
packages at all. However it won't work, since even if all debs are
uploaded, and aptly_add_to_repo is called, debs are moved from temporary
directory to the repository. The proxy may drop connection during this
moment due to stale connection. And just after that we call
aptly_delete_dir which will delete the temporary directory. On the
server side, aptly will however continue to move debs from temporary
directory. Repository is published, but since the temporary directory is
removed during the transaction, the transaction never completely
succeeds. So only some of the debs are uploaded practically. This is now
fixed by adding a sleep of 180s (should be more than enough for all
stuff)
2022-04-14 17:26:22 +05:30
Henrik Grimler 369db121c1
CI: packages: fix if statement syntax .. again 2022-04-14 09:06:08 +02:00
Henrik Grimler 1c71b0db0d
CI: packages: fix if statement syntax
The script uses bash, so need a closing fi statement.
2022-04-14 08:42:07 +02:00
Yaksh Bariya 5b8b15e7aa
fixup(ci:upload): Treat failure to add debs to repository as error 2022-04-14 11:45:17 +05:30
Yaksh Bariya 8b8cd21cd9
respect '%ci:no-build' also for PRs
Helpful when a large dependency rebuild is being done and changes are tested locally or on fork.
2022-04-13 17:38:13 +05:30
Yaksh Bariya 50a98af858
CI: Upload build artifacts even on build failure
Helpful when CI times out or building a large number of packages for testing and it fails
2022-03-30 08:52:01 +05:30
Aditya Alok 92ea572ede
ci(package_updates.yml): use `contact@termux.org` for git email
Signed-off-by: Aditya Alok <dev.aditya.alok@gmail.com>
2022-03-28 23:45:07 +05:30
Aditya Alok 5377dbc8b8 ci(package_updates.yml): adhere to new update system
Signed-off-by: Aditya Alok <dev.aditya.alok@gmail.com>
2022-03-28 23:11:11 +05:30
Henrik Grimler e585e63454
CI: upload to packages.termux.org again
The cloudflare variant, packages-cf.termux.org, doesn't like large
uploads (> 100M?).
2022-03-25 17:43:25 +01:00
Henrik Grimler 829368cac9
CI: set GPG_PASSHRASE as well for packages.termux.org uploads 2022-03-13 13:44:05 +01:00
Henrik Grimler 3997a41ff0
CI: remove duplicate info message
It's already printed in aptly_publish_repo.
2022-03-13 13:28:22 +01:00
Henrik Grimler e7be93609b
CI: extract deb archive only once
No need to extract it again during grimler.se upload.
2022-03-13 13:15:16 +01:00
Henrik Grimler ccf1b2d111
CI: packages.yml: upload to grimler.se as well 2022-03-13 11:12:42 +01:00
Henrik Grimler e7f3a436ae
CI: packages.yml: refactor, move aptly api functions to scripts/ 2022-03-13 11:12:39 +01:00
Tee KOBAYASHI 8a5fe78e76 github actions: Properly build custom Docker image
when `setup-android-sdk.sh` is modified.
2022-03-13 08:30:29 +09:00
Henrik Grimler a429f53d6f
CI: use cloudflare url for uploading packages
Since the fosshost proxy does not work at the moment.
2022-03-07 11:15:48 +01:00
Henrik Grimler eb66dbe60f
package_updates: switch contact email to one at termux.org 2022-02-18 11:01:58 +01:00
Henrik Grimler a5994b702c
workflows: docker: change auth user, and use docker/login-action
This should make automatic updates work again. docker/login-action
allows for authentication with a token instead of password.

Inspiration taken from workflow in github.com/termux/termux-docker.
2022-02-08 10:23:05 +01:00
Yaksh Bariya 9874a3fd8d
Revert "Delete Package Update workflow" 2022-02-07 17:02:43 +05:30
Yaksh Bariya f6b5153ac9
Delete Package Update workflow 2022-02-07 17:02:31 +05:30
Henrik Grimler 567e223183
Revert "github actions: use per-repository token for automatic package updates workflow"
This reverts commit fc95ff7881.

Unfortunately GITHUB_TOKEN authentication cannot start other action
(recent automatic update pushes haven't been built), so switch back to
a new personal access token (generated by me).

Ref: https://stackoverflow.com/a/67551255
2022-02-02 14:14:57 +01:00
xeffyr fc95ff7881
github actions: use per-repository token for automatic package updates workflow 2022-01-31 19:47:26 +02:00
Henrik Grimler ca5438c1ef
CI: fix occasional dependency upload issue
Previously we were outputting built debs into ./debs instead of
standard folder ./output.  We were also sorting built_packages.txt so
that $(cat ./built_packages.txt) returned an alphabetically sorted
list.

This caused issues for pushes that updated both a package and one or
more of its dependencies, if the dependencies start with a letter that
is later in the alphabet.

Latest example was libforestdb and libsnappy.  Libforestdb depends on
libsnappy, but the buildorder as given to build-package.sh was
"libforestdb libsnappy".  During the libforestdb build libsnappy was
built first, but the deb put into the standard ./output folder (that's
how our build system currently handles built dependency debs).  When
build-package.sh then later tries building libsnappy it skips it
since it has already been built.  We then end up with only libforestdb
in the ./debs folder.

Fix this issue by putting all debs in ./output/, and then moving the
once mentioned in built_packages.txt to ./debs/.

Fixes https://github.com/termux/termux-packages/issues/8773.
2022-01-30 17:06:27 +01:00
Leonid Pliushch 88b2e3efd4
github actions: set --max-time option for curl calls where it can hang for long time 2022-01-15 13:12:39 +02:00
Leonid Pliushch 750a529467
github actions: filter http codes in other places, not only when publishing changes 2021-12-26 22:41:08 +02:00
Leonid Pliushch d332ec887e
github actions: change regexp for packages
Allow dot in package name.
2021-12-20 17:21:55 +02:00
Leonid Pliushch 7aa03f39d8
github actions: filter http codes when publishing changes, handle 504
Aptly may process request for long time enough to make CloudFlare or
server drop connection. Treat this as false failure and don't stop
GitHub Actions with error status.
2021-12-05 12:03:00 +02:00
Yaksh Bariya 99468e7fa8
Fix typo in packages.yml 2021-11-04 19:59:01 +05:30
Yaksh Bariya f3eae9f980
CI: Also package debs for subpackages when workflow_dispatch 2021-11-04 19:38:17 +05:30
Yaksh Bariya 46423f3675
CI: limit package-updates actions scope
Running this action in forks has literally no sense
2021-10-31 10:33:30 +05:30
Leonid Pliushch 3194a85278
github actions: introduce workflow_dispatch trigger for rebuilding packages 2021-10-26 20:19:10 +03:00
Lucinda May Phipps 80a69bf5e0
packages.yml: allow packages with + in name 2021-10-23 13:40:37 +01:00
Leonid Pliushch 8bde94f6a3
github actions: don't use --retry-all-errors curl option as unavailable in this version 2021-10-09 00:24:54 +03:00
Leonid Pliushch 23ba08388b
github actions: get rid of retry loops 2021-10-08 23:48:58 +03:00
Leonid Pliushch 160e810be8
github actions: use cloudflare endpoint in all problematic places
Should finally resolve issues with uploads.
2021-10-08 23:36:57 +03:00
Leonid Pliushch b8b89471f5
github actions: upload delays 30s, retries 3
Do less retries but with higher delay. Previous config didn't work.
2021-10-08 23:24:30 +03:00
Leonid Pliushch 5272d228e5
github actions: specify own user agent for all curl commands 2021-10-08 14:05:25 +03:00
Leonid Pliushch df3a651aa8
github actions: package upload delays 10s and 10 retry attempts
Also noticed that planned initial delay of 3s hasn't been implemented at
all.
2021-10-08 14:01:17 +03:00
Leonid Pliushch 938f0e4da6
github actions: don't exit with failure status on publishing
Revert to old behavior: if connection has been dropped, do not exit
with failure status.
2021-10-07 16:43:29 +03:00
Leonid Pliushch 8c7b2298f5
github actions: retry problematic steps up to 5 times 2021-10-07 16:41:21 +03:00
Leonid Pliushch 49e2fe69cd
github actions: fix curl command for publishing step 2021-10-07 15:52:53 +03:00
Leonid Pliushch 9fc7764e09
github actions: mask curl exit code 2021-10-07 15:36:30 +03:00
Leonid Pliushch c4ac6712bc
github actions: manually handle curl exit code 52 2021-10-07 15:08:36 +03:00
Leonid Pliushch 19acaed7cd
github actions: add time tracking to upload logs 2021-10-07 15:02:11 +03:00
Leonid Pliushch 150479d560
github actions: retry up to 3 times when connection has been dropped 2021-10-07 14:49:01 +03:00