Skip to content

Instantly share code, notes, and snippets.

@minamijoyo
Last active March 1, 2024 06:09
Show Gist options
  • Star 21 You must be signed in to star a gist
  • Fork 8 You must be signed in to fork a gist
  • Save minamijoyo/3d8aa79085369efb79964ba45e24bb0e to your computer and use it in GitHub Desktop.
Save minamijoyo/3d8aa79085369efb79964ba45e24bb0e to your computer and use it in GitHub Desktop.
Using GitHubPrivateRepositoryReleaseDownloadStrategy removed in brew v2
require "formula"
require_relative "lib/private_strategy"
class Hoge < Formula
homepage "https://github.com/yourcompany/hoge"
url "https://github.com/yourcompany/hoge/releases/download/v0.1.0/hoge_v0.1.0_darwin_amd64.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy
sha256 "6de411ff3e4b1658a413dd6181fcXXXXXXXXXXXXXXXXXXXX"
head "https://github.com/yourcompany/hoge.git"
version "0.1.0"
def install
bin.install "hoge"
end
end
brew tap yourcompany/tap git@github.com:yourcompany/homebrew-tap.git
export HOMEBREW_GITHUB_API_TOKEN=xxx
brew install hoge
# Save this file as `lib/private_strategy.rb`
# Add `require_relative "lib/private_strategy"` to your formula.
#
# This is based on the following, with minor fixes.
# https://github.com/Homebrew/brew/blob/193af1442f6b9a19fa71325160d0ee2889a1b6c9/Library/Homebrew/compat/download_strategy.rb#L48-L157
# BSD 2-Clause License
#
# Copyright (c) 2009-present, Homebrew contributors
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# * Redistributions of source code must retain the above copyright notice, this
# list of conditions and the following disclaimer.
#
# * Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# GitHubPrivateRepositoryDownloadStrategy downloads contents from GitHub
# Private Repository. To use it, add
# `:using => GitHubPrivateRepositoryDownloadStrategy` to the URL section of
# your formula. This download strategy uses GitHub access tokens (in the
# environment variables `HOMEBREW_GITHUB_API_TOKEN`) to sign the request. This
# strategy is suitable for corporate use just like S3DownloadStrategy, because
# it lets you use a private GitHub repository for internal distribution. It
# works with public one, but in that case simply use CurlDownloadStrategy.
class GitHubPrivateRepositoryDownloadStrategy < CurlDownloadStrategy
require "utils/formatter"
require "utils/github"
def initialize(url, name, version, **meta)
super
parse_url_pattern
set_github_token
end
def parse_url_pattern
unless match = url.match(%r{https://github.com/([^/]+)/([^/]+)/(\S+)})
raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Repository."
end
_, @owner, @repo, @filepath = *match
end
def download_url
"https://#{@github_token}@github.com/#{@owner}/#{@repo}/#{@filepath}"
end
private
def _fetch(url:, resolved_url:)
curl_download download_url, to: temporary_path
end
def set_github_token
@github_token = ENV["HOMEBREW_GITHUB_API_TOKEN"]
unless @github_token
raise CurlDownloadStrategyError, "Environmental variable HOMEBREW_GITHUB_API_TOKEN is required."
end
validate_github_repository_access!
end
def validate_github_repository_access!
# Test access to the repository
GitHub.repository(@owner, @repo)
rescue GitHub::HTTPNotFoundError
# We only handle HTTPNotFoundError here,
# becase AuthenticationFailedError is handled within util/github.
message = <<~EOS
HOMEBREW_GITHUB_API_TOKEN can not access the repository: #{@owner}/#{@repo}
This token may not have permission to access the repository or the url of formula may be incorrect.
EOS
raise CurlDownloadStrategyError, message
end
end
# GitHubPrivateRepositoryReleaseDownloadStrategy downloads tarballs from GitHub
# Release assets. To use it, add
# `:using => GitHubPrivateRepositoryReleaseDownloadStrategy` to the URL section of
# your formula. This download strategy uses GitHub access tokens (in the
# environment variables HOMEBREW_GITHUB_API_TOKEN) to sign the request.
class GitHubPrivateRepositoryReleaseDownloadStrategy < GitHubPrivateRepositoryDownloadStrategy
def initialize(url, name, version, **meta)
super
end
def parse_url_pattern
url_pattern = %r{https://github.com/([^/]+)/([^/]+)/releases/download/([^/]+)/(\S+)}
unless @url =~ url_pattern
raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Release."
end
_, @owner, @repo, @tag, @filename = *@url.match(url_pattern)
end
def download_url
"https://#{@github_token}@api.github.com/repos/#{@owner}/#{@repo}/releases/assets/#{asset_id}"
end
private
def _fetch(url:, resolved_url:)
# HTTP request header `Accept: application/octet-stream` is required.
# Without this, the GitHub API will respond with metadata, not binary.
curl_download download_url, "--header", "Accept: application/octet-stream", to: temporary_path
end
def asset_id
@asset_id ||= resolve_asset_id
end
def resolve_asset_id
release_metadata = fetch_release_metadata
assets = release_metadata["assets"].select { |a| a["name"] == @filename }
raise CurlDownloadStrategyError, "Asset file not found." if assets.empty?
assets.first["id"]
end
def fetch_release_metadata
release_url = "https://api.github.com/repos/#{@owner}/#{@repo}/releases/tags/#{@tag}"
GitHub.open_api(release_url)
end
end
@ed9w2in6
Copy link

Hi, just to let you know, this helped my a lot. Much thanks!

@yoavgeva
Copy link

Helped me a lot! thanks!

@danieldonoghue
Copy link

hi :)

any chance of an update for this? getting a deprecation warning for _fetch()

@ed9w2in6
Copy link

ed9w2in6 commented Apr 25, 2021

Well, I don't know how to fix the WARNING with _fetch(), but I do know the line GitHub.open_api(release_url) needs to be replaced by GitHub::API.open_rest(release_url, data: nil, data_binary_path: nil, request_method: nil, scopes: [].freeze, parse_json: true) due to an ERROR.

I chose the arguments to match the default values for the open_api() function as of now @2021 April 26 see: this

This is because homebrew had disabled it for some reason. Try removing your caches and reinstall your private formula, it will complain:

Error: Calling GitHub.open_api is disabled! Use GitHub::API.open_rest instead.

@ZPascal
Copy link

ZPascal commented Jun 9, 2021

Hi @danieldonoghue, I have the same problem. Have you found a solution in the meantime?

@ZPascal
Copy link

ZPascal commented Jun 16, 2021

Hi community and @danieldonoghue, I fixed the deprecation warning by adding the timeout parameter to the function.

def _fetch(url:, resolved_url:, timeout:)
 curl_download download_url, to: temporary_path  
end
def _fetch(url:, resolved_url:, timeout:)    
# HTTP request header `Accept: application/octet-stream` is required.    
# Without this, the GitHub API will respond with metadata, not binary.    
 curl_download download_url, "--header", "Accept: application/octet-stream", to: temporary_path  
end

After the update of the functionality, homebrew use the current version of the functions (CurlPostDownloadStrategy | CurlDownloadStrategy) and not the deprecated functionality.

@minamijoyo Is there the possibility to update your code, include my fix and fix this deprecation warning?

@xavidop
Copy link

xavidop commented Jun 22, 2021

Hi community and @danieldonoghue, I fixed the deprecation warning by adding the timeout parameter to the function.

def _fetch(url:, resolved_url:, timeout:)
 curl_download download_url, to: temporary_path  
end
def _fetch(url:, resolved_url:, timeout:)    
# HTTP request header `Accept: application/octet-stream` is required.    
# Without this, the GitHub API will respond with metadata, not binary.    
 curl_download download_url, "--header", "Accept: application/octet-stream", to: temporary_path  
end

After the update of the functionality, homebrew use the current version of the functions (CurlPostDownloadStrategy | CurlDownloadStrategy) and not the deprecated functionality.

@minamijoyo Is there the possibility to update your code, include my fix and fix this deprecation warning?

thanks for the fix!!! now my tap is working again

@minamijoyo
Copy link
Author

Hi, all! Sorry for the late reply. Let me share my context about this gist.

I'm an original author of the patch of this feature, the GitHubPrivateRepositoryReleaseDownloadStrategy. (Homebrew/brew#1763)

One day, I noticed that the brew v2 removed the feature, so I salvaged the last implementation under the brew's license before removed for whom anyone need this feature.

Time goes by... The upstream has changed and some code seems to be broken now as you reported. However, unfortunately, I'm no longer using this feature for now and I have no motivation about this gist up-to-date. I understand someone still need this feature, so my proposal for whom anyone interested in this is creating a new repository to maintain this feature up-to-date by the community. Feel free to fork it.

Thanks!

@ZPascal
Copy link

ZPascal commented Jun 23, 2021

I have forked the corresponding gist and changed the corresponding parts.

@Frizlab
Copy link

Frizlab commented Dec 29, 2021

It does not seem to work anymore, sadly.
I get an error: Download failed on Cask 'MyCask' with message: unknown keyword: timeout.

@ZPascal
Copy link

ZPascal commented Dec 29, 2021

@Frizlab You can check out the working fork.

@Frizlab
Copy link

Frizlab commented Dec 29, 2021

Thanks!

@guilhermeprokisch
Copy link

I'm getting Invalid url pattern for GitHub Release. with the new release URL format

@ZPascal
Copy link

ZPascal commented Feb 14, 2023

@guilhermeprokisch Can you please share with us the corresponding error message? Have you already checked out the maintained fork?

@lRoMYl
Copy link

lRoMYl commented Apr 11, 2023

Hey guys, recently it seems that the formulae just ignored the defined custom strategy entirely

require_relative "lib/private_strategy"

class MyCustomFormula < Formula
  url "https://github.com/{repo-owner}/{repo-name}/archive/0.5.6.tar.gz", :using => GitHubPrivateRepositoryDownloadStrategy
  sha256 "{sha256}"
  head "https://github.com/{repo-owner}{repo-name}.git"

  depends_on :xcode => :build
  depends_on :macos

  def install
    system "make", "install", "PREFIX=#{prefix}"
  end
end

When using debug mode, it seems to be calling the cURL and overrides the the URL and headers automatically without using the download_url format defined in this script: "https://#{@github_token}@github.com/#{@owner}/#{@repo}/#{@filepath}"

I suspect the latest homebrew hardcoded the github link to be downloaded using a custom strategy and found a couple possible candidates here but I'm not familiar enough with Ruby to overwrite this default behaviour.
https://github.com/Homebrew/brew/search?q=vnd.github&type=code
Over here , it seems like download_strategy has a strategy to decide the download strategy to be used
https://github.com/Homebrew/brew/blob/d15f571eb6a994e7ca689721a54c5c6bff4219a0/Library/Homebrew/download_strategy.rb

Anyone have a good idea on how to fix this?

/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.11-130-gfc7eaab\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.5\)\ curl/7.79.1 --header Accept-Language:\ en --retry 3 --location https://api.github.com/repos/{repo-owner}/{repo-name}--header Accept:\ application/vnd.github\+json --write-out '
'\%\{http_code\} --header Authorization:\ token\ ****** --header X-GitHub-Api-Version:2022-11-28 --dump-header /private/tmp/github_api_headers20230411-83133-13py20

/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.11-130-gfc7eaab\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.5\)\ curl/7.79.1 --header Accept-Language:\ en --retry 3 --fail --location --silent --head https://github.com/{repo-owner}/{repo-name}/archive/0.5.6.tar.gz

/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.11-130-gfc7eaab\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.5\)\ curl/7.79.1 --header Accept-Language:\ en --retry 3 --fail --location --silent --head --request GET https://github.com/{repo-owner}/{repo-name}/archive/0.5.6.tar.gz

@xavidop
Copy link

xavidop commented Apr 11, 2023

we are facing the same issue as @lRoMYl

@mike-carey
Copy link

This issue seems to be the culprit. There is a head request to the browser download url that does not have the token. I was able to workaround this by overriding the resolve_url_basename_time_file_size:

  def resolve_url_basename_time_file_size(url, timeout: nil)
    url = download_url
    super
  end

@xavidop
Copy link

xavidop commented Apr 11, 2023

it worked for us!! thanks @mike-carey

@lRoMYl
Copy link

lRoMYl commented Apr 12, 2023

Worked for us too, thanks @mike-carey

@Ronald-TR
Copy link

Worked for me! Thanks @mike-carey !!

@misterdorm
Copy link

+1, appreciate it @mike-carey . I spent a couple hours going down this rathole troubleshooting and finally ended up here!

@ed9w2in6
Copy link

ed9w2in6 commented Oct 14, 2023

Hi lads, hope this will be helpful for you if you are still using this strategy:

TL;DR you will need to update your strategy to override

both resolve_url_basename_time_file_size and resolved_basename** NEW
to address all use cases (see next section):

class GitHubPrivateRepositoryDownloadStrategy < CurlDownloadStrategy
  require "utils/formatter"
  require "utils/github"

  # fix issue: https://github.com/Homebrew/brew/issues/15169
  # bypass a HEAD request that does NOT contains token, which will fail
  def resolve_url_basename_time_file_size(url, timeout: nil)
    url = download_url
    super
  end

  # [2023-10-14] brew relies on this output to rename the downloaded file
  # See: https://github.com/Homebrew/brew/blob/fbe50bf280bff033b968d439d5441d338afec98f/Library/Homebrew/download_strategy.rb#L305
  # Not setting this will break formulas during install stage, symtoms: Errno::ENOENT: No such file or directory - path/to/file
  def resolved_basename
    @filename
  end
# ... skipped

I think at this point everyone should have overridden their resolve_url_basename_time_file_size method to deal
with the forced HEAD request as mentioned at Homebrew/brew#15169.

However, please read this the definition of cached_location:
https://github.com/Homebrew/brew/blob/fbe50bf280bff033b968d439d5441d338afec98f/Library/Homebrew/download_strategy.rb#L305

In short, AFAIK the practice in Homebrew is all download strategy should download stuff to temporary_path. The dependency is as follows:
temporary_path -> resolved_basename -> resolved_url_and_basename -> resolve_url_basename_time_file_size (for CurlDownloadStrategy only)

From this hierarchy we can see that if we overridden the former it will break download path too.
Symtoms are:

(during call to InstallRenamed.install_p)

Errno::ENOENT: No such file or directory - path/to/file

Of course an alternative is to just override resolve_url_basename_time_file_size with:

# from https://github.com/Homebrew/brew/issues/15169#issuecomment-1500653530
def resolve_url_basename_time_file_size(url, timeout: nil)
  [download_url, "", Time.now, 0, false]
end

However this will be less robust to changes to Homebrew's code.


Affected use case

From what I know this issue seems to only affect formulas that just installs a file via the bin.install directive.
For archives, my guess is extraction will be carried out transparently first, hence the file names inside the archive will be preserved.
I have NOT drill into how this the exact process is tho, so please share if you did.

However, for consistency sake I will suggest everyone to make this change.
Changing should also guard against behaviour changes on Homebrew's side, say if they decide to suffix all extracted files for whatever reasons.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment