mbox to Google Groups

mbox2gg.py

#
# Import an mbox file into Google Groups using the Groups Migration API
#
# To run this script, you will need to:
#     1. Be an admin on your domain
#     2. Enable API access in your domain
#     3. Create a project on the developer console
#     4. Activate the Groups Migration API for the project
#     5. Create an OAuth Client ID and Secret for the project
#     6. Install the google-api-python-client and google-auth-oauthlib 
#        packages into your python environment (tested in python 3)
#
# Messy Details (recorded 2018-07-02):
# 2. See https://developers.google.com/admin-sdk/groups-migration/v1/guides/prerequisites
# 3. Go to https://console.developers.google.com
#       Create a new project
#           Pick a meaningful name and set the Organization and Location to your domain name
# 4. Select your project in the developer console (https://console.developers.google.com)
#        Go the the dashboard page and click "Enable APIs and Servces"
#        Find the Groups Migration API and click it
#        Click the "Enable" button
# 5. Select your project in the developer console (https://console.developers.google.com)
#        Go to the credentials page/tab and click "Create Credentials"
#        Choose OAuth Client ID
#        Click "Configure Consent Screen"
#        Set a product name. (This will appear in an authorization prompt when you attempt to run the script.)
#        Click "Save"
#        Choose "Other" for the application type and choose a name (e.g. "mbox2gg") for the OAuth client ID
#        Click "Create"
#        Your OAuth Client ID should now be listed on the Credentials page/tab. Click the download link
#            to save the json file.
#
# References:
#    Groups Migration API:
#        https://developers.google.com/admin-sdk/groups-migration/v1/get-start/getting-started
#    Python API Client:
#        https://developers.google.com/api-client-library/python/
#    Google Auth / OAuth:
#        http://google-auth-oauthlib.readthedocs.io/en/latest/
#

import mailbox
import time
from io import StringIO
import apiclient                              # pip install --upgrade google-api-python-client
from google_auth_oauthlib.flow import Flow    # pip install --upgrade google-auth-oauthlib


# File containing the OAuth Client ID / Secret
client_secret_file = 'client_secret.json'

# The email address of the group to import to (e.g. "groupname@example.com")
groupId = input('Enter groupId: ')

# The mbox file you want to import
mbox_path = input('Enter mbox_path: ')


# Scope for the Groups Migration API
# See https://developers.google.com/admin-sdk/groups-migration/v1/guides/authorizing
scope = 'https://www.googleapis.com/auth/apps.groups.migration'

# 
# The next few lines to handle authentication are taken almost verbatim from
# http://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html
#
# The original version of this script used the oauth2client library (which is now [2018-07-02]
# deprecated) for authentication and included a feature for reading saved credentials from a 
# file. The google_auth_oauthlib library doesn't have comparable functionality (and I haven't
# spent the time to figure out how to do it) so this script will prompt for authorization every
# time it runs.
#

# Create the flow using the client secrets file from the Google API Console.
flow = Flow.from_client_secrets_file(
    client_secret_file,
    scopes=[scope],
    redirect_uri='urn:ietf:wg:oauth:2.0:oob')

# Tell the user to go to the authorization URL.
auth_url, _ = flow.authorization_url(prompt='consent')

print('Please go to this URL: {}'.format(auth_url))

# The user will get an authorization code. This code is used to get the
# access token.
code = input('Enter the authorization code: ')
flow.fetch_token(code=code)



# Can now get the credentials after the call to fetch_token()
credentials = flow.credentials

# Create an object for interacting with the Groups Migration API
#service = discovery.build('groupsmigration', 'v1', credentials=credentials)
service = apiclient.discovery.build('groupsmigration', 'v1', credentials=credentials)

# Open the mbox file
mb = mailbox.mbox(mbox_path)  # The path of the mbox file to import

# Create a counter variable and find the total number of messages
# Use this for status updates as messages are uploaded
i = 1
total_messages = len(mb)

# Process messages from the mbox file one-at-a-time
for msg in mb:
    # Read the rfc822 text (i.e. the message) as a stream object
    stream = StringIO()
    stream.write(msg.as_string())
    
    # Create an object/description suitable for uploading
    media = apiclient.http.MediaIoBaseUpload(stream, mimetype='message/rfc822')
    
    # Upload message to the group archive and get the response
    response = service.archive().insert(groupId=groupId, media_body=media).execute()

    # Show a status message
    print('Message {} of {}: {}'.format(
        i,
        total_messages,
        response['responseCode'])
    )

    i = i + 1

    # Limit to no more than 10 messages per second to avoid exceeding API quota
    # https://developers.google.com/admin-sdk/groups-migration/v1/limits
    time.sleep(0.1)

print('Done.')

@evalinde thank you so much for your support.

Now I get run the Script but I receive a lot of errors when it's migrate, I send you the error

Traceback (most recent call last):
  File "mboxgg.py", line 111, in <module>
    stream.write(msg.as_string())
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\email\message.py", line 158, in as_string
    g.flatten(self, unixfrom=unixfrom)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\email\generator.py", line 116, in flatten
    self._write(msg)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\email\generator.py", line 181, in _write
    self._dispatch(msg)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\email\generator.py", line 214, in _dispatch
    meth(msg)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\email\generator.py", line 272, in _handle_multipart
    g.flatten(part, unixfrom=False, linesep=self._NL)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\email\generator.py", line 116, in flatten
    self._write(msg)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\email\generator.py", line 181, in _write
    self._dispatch(msg)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\email\generator.py", line 214, in _dispatch
    meth(msg)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\email\generator.py", line 243, in _handle_text
    msg.set_payload(payload, charset)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\email\message.py", line 315, in set_payload
    payload = payload.encode(charset.output_charset)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 3622-3623: ordinal not in range(128)

Thanks for your support!!!!

Hi Evalinde,

Thanks you su much for your Script, for me it was very usefull but now I have a problem and I don't understand Why. Your Script in Windows I dont have problems but when I use my Mac (Big Sur MacBook Pro (13-inch, M1, 2020) I receive this error;

I have installed python Version Python 2.7.18 and this is my screenshot where I have my MboxFile and your Script.

Could you help me?

Many thanks

I think you've missed the important part of the error in your screenshot -- it should be the message immediately below the caret character pointing to the @ sign.

But before worrying about that, try python 3. (I don't know if this script is compatible with python 2, but I know I haven't used it with python 2.)

I send you my error

jacobogarrido@MacBook-Pro MboxGroups % python3 mboxggup.py
Traceback (most recent call last):
File "/Users/jacobogarrido/MboxGroups/mboxggup.py", line 45, in
import apiclient # pip install --upgrade google-api-python-client
ModuleNotFoundError: No module named 'apiclient'
jacobogarrido@MacBook-Pro MboxGroups % pip install --upgrade google-api-python-client
Traceback (most recent call last):
File "/usr/local/bin/pip", line 11, in
load_entry_point('pip==21.0.1', 'console_scripts', 'pip')()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py", line 2843, in load_entry_point
return ep.load()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py", line 2434, in load
return self.resolve()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py", line 2440, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/Library/Python/2.7/site-packages/pip-21.0.1-py2.7.egg/pip/_internal/cli/main.py", line 60
sys.stderr.write(f"ERROR: {exc}")

I installed homebrew in my mac and I don't know where is the next step to solve this issue.

Many thanks

I fixed it. I put the command pip3 install --upgrade google-api-python-client and run perfectly.

Thank you so much for your support.

Kind Regards !!!

Hello,
I tried this code and for some messages the import worked, but for most of my messages i got the following error:
<HttpError 400 when requesting https://groupsmigration.googleapis.com/upload/groups/v1/groups/orla.baumgartner%40steadysense.at/archive?alt=json&uploadType=media returned "Unable to parse the raw message". Details: "[{'message': 'Unable to parse the raw message', 'domain': 'global', 'reason': 'invalid'}]">

What is the problem with my emails, and how can I resolve this error?

Thanks in advance.