-
-
Save minamijoyo/3d8aa79085369efb79964ba45e24bb0e to your computer and use it in GitHub Desktop.
require "formula" | |
require_relative "lib/private_strategy" | |
class Hoge < Formula | |
homepage "https://github.com/yourcompany/hoge" | |
url "https://github.com/yourcompany/hoge/releases/download/v0.1.0/hoge_v0.1.0_darwin_amd64.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy | |
sha256 "6de411ff3e4b1658a413dd6181fcXXXXXXXXXXXXXXXXXXXX" | |
head "https://github.com/yourcompany/hoge.git" | |
version "0.1.0" | |
def install | |
bin.install "hoge" | |
end | |
end |
brew tap yourcompany/tap git@github.com:yourcompany/homebrew-tap.git | |
export HOMEBREW_GITHUB_API_TOKEN=xxx | |
brew install hoge |
# Save this file as `lib/private_strategy.rb` | |
# Add `require_relative "lib/private_strategy"` to your formula. | |
# | |
# This is based on the following, with minor fixes. | |
# https://github.com/Homebrew/brew/blob/193af1442f6b9a19fa71325160d0ee2889a1b6c9/Library/Homebrew/compat/download_strategy.rb#L48-L157 | |
# BSD 2-Clause License | |
# | |
# Copyright (c) 2009-present, Homebrew contributors | |
# All rights reserved. | |
# | |
# Redistribution and use in source and binary forms, with or without | |
# modification, are permitted provided that the following conditions are met: | |
# | |
# * Redistributions of source code must retain the above copyright notice, this | |
# list of conditions and the following disclaimer. | |
# | |
# * Redistributions in binary form must reproduce the above copyright notice, | |
# this list of conditions and the following disclaimer in the documentation | |
# and/or other materials provided with the distribution. | |
# | |
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" | |
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE | |
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | |
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE | |
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL | |
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR | |
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER | |
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, | |
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE | |
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | |
# GitHubPrivateRepositoryDownloadStrategy downloads contents from GitHub | |
# Private Repository. To use it, add | |
# `:using => GitHubPrivateRepositoryDownloadStrategy` to the URL section of | |
# your formula. This download strategy uses GitHub access tokens (in the | |
# environment variables `HOMEBREW_GITHUB_API_TOKEN`) to sign the request. This | |
# strategy is suitable for corporate use just like S3DownloadStrategy, because | |
# it lets you use a private GitHub repository for internal distribution. It | |
# works with public one, but in that case simply use CurlDownloadStrategy. | |
class GitHubPrivateRepositoryDownloadStrategy < CurlDownloadStrategy | |
require "utils/formatter" | |
require "utils/github" | |
def initialize(url, name, version, **meta) | |
super | |
parse_url_pattern | |
set_github_token | |
end | |
def parse_url_pattern | |
unless match = url.match(%r{https://github.com/([^/]+)/([^/]+)/(\S+)}) | |
raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Repository." | |
end | |
_, @owner, @repo, @filepath = *match | |
end | |
def download_url | |
"https://#{@github_token}@github.com/#{@owner}/#{@repo}/#{@filepath}" | |
end | |
private | |
def _fetch(url:, resolved_url:) | |
curl_download download_url, to: temporary_path | |
end | |
def set_github_token | |
@github_token = ENV["HOMEBREW_GITHUB_API_TOKEN"] | |
unless @github_token | |
raise CurlDownloadStrategyError, "Environmental variable HOMEBREW_GITHUB_API_TOKEN is required." | |
end | |
validate_github_repository_access! | |
end | |
def validate_github_repository_access! | |
# Test access to the repository | |
GitHub.repository(@owner, @repo) | |
rescue GitHub::HTTPNotFoundError | |
# We only handle HTTPNotFoundError here, | |
# becase AuthenticationFailedError is handled within util/github. | |
message = <<~EOS | |
HOMEBREW_GITHUB_API_TOKEN can not access the repository: #{@owner}/#{@repo} | |
This token may not have permission to access the repository or the url of formula may be incorrect. | |
EOS | |
raise CurlDownloadStrategyError, message | |
end | |
end | |
# GitHubPrivateRepositoryReleaseDownloadStrategy downloads tarballs from GitHub | |
# Release assets. To use it, add | |
# `:using => GitHubPrivateRepositoryReleaseDownloadStrategy` to the URL section of | |
# your formula. This download strategy uses GitHub access tokens (in the | |
# environment variables HOMEBREW_GITHUB_API_TOKEN) to sign the request. | |
class GitHubPrivateRepositoryReleaseDownloadStrategy < GitHubPrivateRepositoryDownloadStrategy | |
def initialize(url, name, version, **meta) | |
super | |
end | |
def parse_url_pattern | |
url_pattern = %r{https://github.com/([^/]+)/([^/]+)/releases/download/([^/]+)/(\S+)} | |
unless @url =~ url_pattern | |
raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Release." | |
end | |
_, @owner, @repo, @tag, @filename = *@url.match(url_pattern) | |
end | |
def download_url | |
"https://#{@github_token}@api.github.com/repos/#{@owner}/#{@repo}/releases/assets/#{asset_id}" | |
end | |
private | |
def _fetch(url:, resolved_url:) | |
# HTTP request header `Accept: application/octet-stream` is required. | |
# Without this, the GitHub API will respond with metadata, not binary. | |
curl_download download_url, "--header", "Accept: application/octet-stream", to: temporary_path | |
end | |
def asset_id | |
@asset_id ||= resolve_asset_id | |
end | |
def resolve_asset_id | |
release_metadata = fetch_release_metadata | |
assets = release_metadata["assets"].select { |a| a["name"] == @filename } | |
raise CurlDownloadStrategyError, "Asset file not found." if assets.empty? | |
assets.first["id"] | |
end | |
def fetch_release_metadata | |
release_url = "https://api.github.com/repos/#{@owner}/#{@repo}/releases/tags/#{@tag}" | |
GitHub.open_api(release_url) | |
end | |
end |
Helped me a lot! thanks!
hi :)
any chance of an update for this? getting a deprecation warning for _fetch()
Well, I don't know how to fix the WARNING with _fetch()
, but I do know the line GitHub.open_api(release_url)
needs to be replaced by GitHub::API.open_rest(release_url, data: nil, data_binary_path: nil, request_method: nil, scopes: [].freeze, parse_json: true)
due to an ERROR.
I chose the arguments to match the default values for the open_api()
function as of now @2021 April 26 see: this
This is because homebrew had disabled it for some reason. Try removing your caches and reinstall your private formula, it will complain:
Error: Calling GitHub.open_api is disabled! Use GitHub::API.open_rest instead.
Hi @danieldonoghue, I have the same problem. Have you found a solution in the meantime?
Hi community and @danieldonoghue, I fixed the deprecation warning by adding the timeout parameter to the function.
def _fetch(url:, resolved_url:, timeout:)
curl_download download_url, to: temporary_path
end
def _fetch(url:, resolved_url:, timeout:)
# HTTP request header `Accept: application/octet-stream` is required.
# Without this, the GitHub API will respond with metadata, not binary.
curl_download download_url, "--header", "Accept: application/octet-stream", to: temporary_path
end
After the update of the functionality, homebrew use the current version of the functions (CurlPostDownloadStrategy | CurlDownloadStrategy) and not the deprecated functionality.
@minamijoyo Is there the possibility to update your code, include my fix and fix this deprecation warning?
Hi community and @danieldonoghue, I fixed the deprecation warning by adding the timeout parameter to the function.
def _fetch(url:, resolved_url:, timeout:) curl_download download_url, to: temporary_path enddef _fetch(url:, resolved_url:, timeout:) # HTTP request header `Accept: application/octet-stream` is required. # Without this, the GitHub API will respond with metadata, not binary. curl_download download_url, "--header", "Accept: application/octet-stream", to: temporary_path endAfter the update of the functionality, homebrew use the current version of the functions (CurlPostDownloadStrategy | CurlDownloadStrategy) and not the deprecated functionality.
@minamijoyo Is there the possibility to update your code, include my fix and fix this deprecation warning?
thanks for the fix!!! now my tap is working again
Hi, all! Sorry for the late reply. Let me share my context about this gist.
I'm an original author of the patch of this feature, the GitHubPrivateRepositoryReleaseDownloadStrategy. (Homebrew/brew#1763)
One day, I noticed that the brew v2 removed the feature, so I salvaged the last implementation under the brew's license before removed for whom anyone need this feature.
Time goes by... The upstream has changed and some code seems to be broken now as you reported. However, unfortunately, I'm no longer using this feature for now and I have no motivation about this gist up-to-date. I understand someone still need this feature, so my proposal for whom anyone interested in this is creating a new repository to maintain this feature up-to-date by the community. Feel free to fork it.
Thanks!
I have forked the corresponding gist and changed the corresponding parts.
It does not seem to work anymore, sadly.
I get an error: Download failed on Cask 'MyCask' with message: unknown keyword: timeout
.
Thanks!
I'm getting Invalid url pattern for GitHub Release. with the new release URL format
@guilhermeprokisch Can you please share with us the corresponding error message? Have you already checked out the maintained fork?
Hey guys, recently it seems that the formulae just ignored the defined custom strategy entirely
require_relative "lib/private_strategy"
class MyCustomFormula < Formula
url "https://github.com/{repo-owner}/{repo-name}/archive/0.5.6.tar.gz", :using => GitHubPrivateRepositoryDownloadStrategy
sha256 "{sha256}"
head "https://github.com/{repo-owner}{repo-name}.git"
depends_on :xcode => :build
depends_on :macos
def install
system "make", "install", "PREFIX=#{prefix}"
end
end
When using debug mode, it seems to be calling the cURL and overrides the the URL and headers automatically without using the download_url format defined in this script: "https://#{@github_token}@github.com/#{@owner}/#{@repo}/#{@filepath}"
I suspect the latest homebrew hardcoded the github link to be downloaded using a custom strategy and found a couple possible candidates here but I'm not familiar enough with Ruby to overwrite this default behaviour.
https://github.com/Homebrew/brew/search?q=vnd.github&type=code
Over here , it seems like download_strategy has a strategy to decide the download strategy to be used
https://github.com/Homebrew/brew/blob/d15f571eb6a994e7ca689721a54c5c6bff4219a0/Library/Homebrew/download_strategy.rb
Anyone have a good idea on how to fix this?
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.11-130-gfc7eaab\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.5\)\ curl/7.79.1 --header Accept-Language:\ en --retry 3 --location https://api.github.com/repos/{repo-owner}/{repo-name}--header Accept:\ application/vnd.github\+json --write-out '
'\%\{http_code\} --header Authorization:\ token\ ****** --header X-GitHub-Api-Version:2022-11-28 --dump-header /private/tmp/github_api_headers20230411-83133-13py20
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.11-130-gfc7eaab\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.5\)\ curl/7.79.1 --header Accept-Language:\ en --retry 3 --fail --location --silent --head https://github.com/{repo-owner}/{repo-name}/archive/0.5.6.tar.gz
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.11-130-gfc7eaab\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.5\)\ curl/7.79.1 --header Accept-Language:\ en --retry 3 --fail --location --silent --head --request GET https://github.com/{repo-owner}/{repo-name}/archive/0.5.6.tar.gz
we are facing the same issue as @lRoMYl
This issue seems to be the culprit. There is a head request to the browser download url that does not have the token. I was able to workaround this by overriding the resolve_url_basename_time_file_size
:
def resolve_url_basename_time_file_size(url, timeout: nil)
url = download_url
super
end
it worked for us!! thanks @mike-carey
Worked for us too, thanks @mike-carey
Worked for me! Thanks @mike-carey !!
+1, appreciate it @mike-carey . I spent a couple hours going down this rathole troubleshooting and finally ended up here!
Hi lads, hope this will be helpful for you if you are still using this strategy:
TL;DR you will need to update your strategy to override
both resolve_url_basename_time_file_size
and resolved_basename
** NEW
to address all use cases (see next section):
class GitHubPrivateRepositoryDownloadStrategy < CurlDownloadStrategy
require "utils/formatter"
require "utils/github"
# fix issue: https://github.com/Homebrew/brew/issues/15169
# bypass a HEAD request that does NOT contains token, which will fail
def resolve_url_basename_time_file_size(url, timeout: nil)
url = download_url
super
end
# [2023-10-14] brew relies on this output to rename the downloaded file
# See: https://github.com/Homebrew/brew/blob/fbe50bf280bff033b968d439d5441d338afec98f/Library/Homebrew/download_strategy.rb#L305
# Not setting this will break formulas during install stage, symtoms: Errno::ENOENT: No such file or directory - path/to/file
def resolved_basename
@filename
end
# ... skipped
I think at this point everyone should have overridden their resolve_url_basename_time_file_size
method to deal
with the forced HEAD request as mentioned at Homebrew/brew#15169.
However, please read this the definition of cached_location
:
https://github.com/Homebrew/brew/blob/fbe50bf280bff033b968d439d5441d338afec98f/Library/Homebrew/download_strategy.rb#L305
In short, AFAIK the practice in Homebrew is all download strategy should download stuff to temporary_path
. The dependency is as follows:
temporary_path
-> resolved_basename
-> resolved_url_and_basename
-> resolve_url_basename_time_file_size
(for CurlDownloadStrategy
only)
From this hierarchy we can see that if we overridden the former it will break download path too.
Symtoms are:
(during call to InstallRenamed.install_p
)
Errno::ENOENT: No such file or directory - path/to/file
Of course an alternative is to just override resolve_url_basename_time_file_size
with:
# from https://github.com/Homebrew/brew/issues/15169#issuecomment-1500653530
def resolve_url_basename_time_file_size(url, timeout: nil)
[download_url, "", Time.now, 0, false]
end
However this will be less robust to changes to Homebrew's code.
Affected use case
From what I know this issue seems to only affect formulas that just installs a file via the bin.install
directive.
For archives, my guess is extraction will be carried out transparently first, hence the file names inside the archive will be preserved.
I have NOT drill into how this the exact process is tho, so please share if you did.
However, for consistency sake I will suggest everyone to make this change.
Changing should also guard against behaviour changes on Homebrew's side, say if they decide to suffix all extracted files for whatever reasons.
Hi, just to let you know, this helped my a lot. Much thanks!