Skip to content

Instantly share code, notes, and snippets.

@larscwallin

larscwallin/sync-media-smil-explainer-text.md

Last active August 27, 2020 08:41
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save larscwallin/30f0ce72e70162079f2d2f84e2c864cb to your computer and use it in GitHub Desktop.
Save larscwallin/30f0ce72e70162079f2d2f84e2c864cb to your computer and use it in GitHub Desktop.
Download ZIP
Raw
sync-media-smil-explainer-text.md

Sync Media Explainer

DEFINITIONS

  • Sync Media Document

    The SMIL document including all the Sync Media extensions.

  • Media Object

    A media resource in the Sync Media Document referenced using a src attribute.

  • Media Object Reference

    A reference to a Media Object in the Sync Media Document. A Media Object Reference may override attributes and parameters defined on the Media Object.

  • Media Param

    Named parameters that can be defined on a Media Object and Media Object Reference to communicate options to the Media Object Renderer

  • Media Object Renderer

    A component used by the Sync Media Player to render Media Objects of one or more media type.

  • Sync Media Player

    A user agent that knows how to process and play back Sync Media Documents.

  • Timeline

    A timeline that lays out Time Containers in a linear mode.

  • Time Container

    Container elements that dictate the playback mode for its child elements.

  • Parallel Time Container

    A Time Container which child elements will all play in parallel.

  • Sequential Time Container

    A Time Container which child elements will all play in sequence.

GENERAL

This specification describes a file format that lets an author orchestrate the playback of a wide variety of media types in a linear timeline. Sync Media files are rendered to a user by a Sync Media aware player.

Even though the Sync Media spec does support the implementation of generic Sync Media Players, it is reasonable to expect that most implementations will be specialized and refer to one or more of the media types in the Sync Media timeline as being "primary".

An example of such an implementation is an EPUB reading system that will consider the EPUB media type as primary, presenting the public ation as the main content and the other media types as complementary.

Another example would be a system focused on video playback. Such a system would consider any video format as primary and build its presentation of the Sync Media timeline focusing on this aspect of the playback.

Relationship to SMIL

Just as EPUB Media Overlays, Sync Media is based on a subset of SMIL 3.0 features (https://www.w3.org/TR/REC-smil/smil30.html). It is also designed to be "backwards compatible" with its sister spec Media Overlays in order for future synergies.

In comparison to "traditional" SMIL, Sync Media Documents are layout agnostic. The layout and region concepts found in SMIL are not present, instead these aspects are left to the Sync Media Player. Thus the Sync Media format and the Sync Media Player can be embedded in systems such as EPUB reading systems etc, where these systems can customize the presentation according to their respective needs.

It is possible that encodings other than XML, such as JSON, could be used to persist and distribute Sync Media Documents.

PACKAGING AND SIDE LOADING

A Sync Media file can be "containarized" to allow the embedding of some or all of the resources that are referenced by its Sync Media Document. The container format used in these cases is the OCF Zip Container (https://www.w3.org/publishing/epub3/epub-ocf.html#sec-container-zip minus EPUB specific semantics). The OCF format has all of the features, such as encryption; signing etc, needed for the current version as well as foreseeable future versions of Sync Media.

Side-loaded Sync Media

One of the objectives of the Sync Media specification is to let a Sync Media Player add synchronized media, such as Media Overlays, to an existing resource without the need to alter the resouce. This will help the adoption of synchronized media features in the accessibility space by enabling "non-destructive" methods to adding additional modes of narration to existing media resources.

SMIL IMPLEMENTATION

SMIL Modules

Even though many of the Modules present in SMIL are out of scope for the Sync Media specifications purposes, the SMIL 3.0 specification also has concepts that can be used without modification.

Note regarding Timing and Syncronisation

The Timing and Synchronization features in SMIL are very comprehensive. For the purposes of this specification the complete set of SMIL's timing features add too much complexity. A simplified extension module, SyncMediaTiming, is therefore added in order to offer more a more tailored solution.

SMIL EXTENSIONS

SMIL already supports adding additional synonyms to its ref element to make documents easier to read. It also supports the addition of extra Media Object Modules that can add attributes to the various existing modules.

Sync Media extends SMIL's Media Objects by adding additional modules with accompanying attributes / params to convey locator refinements, rendering instructions etc. All additional modules and attributes are prefixed with the Sync Media namespace (in the examples "sync:").

Sync Media SyncMediaObjects support Media Fragment selectors (https://www.w3.org/TR/media-frags/) where such are applicable to their media type.

  • SyncMediaObject (media)

    • lang
    • class

  • SyncMediaObjectReference (mediaref)

    • idref
    • selector

  • SyncMediaObjectText (text)

  • SyncMediaObjectAudio (audio)

    • volume
    • pitch
    • pan

  • SyncMediaObjectVideo (video)

    • volume
    • playbackrate

  • SyncMediaObjectEPUB (epub)

  • SyncMediaObjectPDF (pdf)

    • page
    • viewrect
    • nameddest

    NOTE: See https://www.adobe.com/content/dam/acom/en/devnet/acrobat/pdfs/pdf_open_parameters.pdf

  • SyncMediaObjectHTML (html)

  • SyncMediaClipping

    • clipPath

    NOTES: clipPath allows you to specify a clipping path using a SVG path definition. The clipping is applied to the visible region of the Media Object on which it is defined. When combined with the panZoom module the clipPath is applied inside the rect defined by the panZoom attribute.

  • SyncMediaTiming

    • begin
    • end
    • fill

    NOTES: begin and end attribute values can only be simple length units. fill can have the following values, "repeat", "freeze", "reset" and "remove".

  • SyncMediaUserInteraction

    NOTES: Add specific attributes related to interactive playback modes for Media Objects

Linking SMIL documents

A Sync Media SMIL document can be composed of multiple "root sequences" using xlink functionality. This allows for more effective handling of large volume documents as you can divide the in to more manageable fragments.

To limit complexity it is suggested that linking of SMIL documents is only allowed on direct seq ascendants to the body element.

<smil>
    <body>
        <seq xlink:href="part-1.smil" />
        <seq xlink:href="part-2.smil" />
    </body>
</smil>

IMPLEMENTATION PHASES

In order to have time to get validation of the features described in this document from implementors, it is suggested that features are rolled out in phases.

  1. Phase 1

    This phase has feature parity with EPUB Media Overlays. In other words Basic SMIL Media Objects of type text and audio are supported. No additional timing functionality is implemented apart from what is afforded by the Time Containers.

  2. Phase 2

    This phase adds support for the SyncMediaObjects.

  3. Pahse 3

    This phase adds support for SyncMediaTimeline, PanZoom and SyncMediaClipping.

EXAMPLES

EPUB3 with embedded Media Overlay (EPUB 3.2 Media Overlays compatible)

<smil>
    <head>
        <meta property="media:active-class">-epub-media-overlay-active</meta>
        <meta property="media:playback-active-class">-epub-media-overlay-playing</meta>   
    </head>
    <body>        
        <par id="cover" type="cover">
            <audio src="../audio/narr_cover.mp4" clipBegin="0s" clipEnd="8s" />
            <text src="cover.xhtml#title" />
        </par>
        <seq id="part_1" type="part">
            <seq id="chapter_1" type="chapter" textref="chapter_1.xhtml">
                <par>                    
                    <audio src="../audio/chapter_1.mp4" clipBegin="10s" clipEnd="14s" />
                    <text src="chapter_1.xhtml#f000001" />
                </par>
                <par>                    
                    <audio clipBegin="15s" clipEnd="23s" />
                    <text src="chapter_1.xhtml#f000002" />
                </par>        
            </seq>
        </seq>
    </body>
</smil>

Non-destructive EPUB3 Media Overlays

<smil>
    <head>
        <sync:media type="application/zip+epub" id="epub" src="https://ebooks.com/titles/theexcitingbookofwords/theexcitingbookofwords.epub">
            <param name="active-class" value="-epub-media-overlay-active" valuetype="data" />
            <param name="playback-active-class" value="-epub-media-overlay-playing" valuetype="data" />
        </sync:media>
        <sync:media type="audio/mp4" id="narr_cover" src="https://ebooks.com/titles/theexcitingbookofwords/narration_cover.mp4" />
        <sync:media type="audio/mp4" id="narr_c1" src="https://ebooks.com/titles/theexcitingbookofwords/chapter_1.mp4" />        
    </head>
    <body>        
        <par id="cover" type="cover">
            <sync:mediaref idref="narr_cover" clipBegin="0s" clipEnd="8s" />
            <sync:mediaref idref="epub" sync:selector="epubcfi(/6/4![cover]/4[0:67])" />
        </par>
        <seq id="part_1" type="part">
            <seq id="chapter_1" type="chapter">
                <par>                    
                    <sync:mediaref idref="narr_c1" clipBegin="10s" clipEnd="14s" />
                    <sync:mediaref idref="epub" sync:selector="epubcfi(/6/6![chapter1]/4[0:49])" />
                </par>
                <par>                    
                    <sync:mediaref idref="narr_c1" clipBegin="15s" clipEnd="23s" />
                    <sync:mediaref idref="epub" sync:selector="epubcfi(/6/6![chapter1]/4[50:144])" />
                </par>                
            </seq>
        </seq>
    </body>
</smil>

Audiobook with embedded captions and a cover image

<smil>
    <head>      
        <sync:media type="application/lpf+zip" id="audiobook" src="https://audiobooks.com/titles/theexcitingbookofwords/theexcitingbookofwords.lpf" />
    </head>
    <body>        
        <par id="cover" type="cover">
            <image src="https://audiobooks.com/titles/theexcitingbookofwords/cover.png" 
                    alt="Book title in large exciting font. The background is filled with a multi-colored flurry of words set in varied fonts." />
            <sync:mediaref idref="audiobook" sync:selector="t=0, 8" />
            <text>
                The exciting book of words by L Zamenhof. Read by Lorenzo Vallinio
            </text>
        </par>
        <seq id="part_1" type="part">
            <seq id="chapter_1" type="chapter">
                <par id="chapter_1_par1">                    
                    <sync:mediaref id="chapter_1_par1_audio" idref="audio" sync:selector="t=10, 14" />
                    <text id="chapter_1_par1_text">
                        The lazy dog lay sleeping in the tall grass.
                    </text>
                </par>
                <par id="chapter_1_par2">                    
                    <sync:mediaref id="chapter_1_par2_audio" idref="audio" sync:selector="t=15, 23" />
                    <text id="chapter_1_par2_text">
                        Behind a thick tuft of straws the quick brown fox waited, its hindlegs tensed, ready to leap. 
                    </text>
                </par>
                
            </seq>
        </seq>
    </body>
</smil>

Audiobook synchronized with EPUB3

<smil>
    <head>      
        <sync:media type="application/lpf+zip" id="audiobook" src="https://audiobooks.com/titles/theexcitingbookofwords/theexcitingbookofwords.lpf" />
        <sync:media type="application/zip+epub" id="epub" src="https://ebooks.com/titles/theexcitingbookofwords/theexcitingbookofwords.epub" />
    </head>
    <body>        
        <par id="cover" type="cover">
            <image  src="https://audiobooks.com/titles/theexcitingbookofwords/cover.png" 
                    alt="Book title in large exciting font. The background is filled with a multi-colored flurry of words set in varied fonts." />
            <mediaref idref="audiobook" sync:selector="t=0, 8" />
            <mediaref idref="epub" sync:selector="epubcfi(/6/4![cover]/4[0:67])" />
        </par>
        <seq id="part_1" type="part">
            <seq id="chapter_1" type="chapter">
                <par>                    
                    <sync:mediaref idref="audiobook" sync:selector="t=10, 14" />
                    <sync:mediaref idref="epub" sync:selector="epubcfi(/6/6![chapter1]/4[0:49])" />
                </par>
            </seq>
        </seq>
    </body>
</smil>

Comicbook example (with guided navigation using panZoom and clipPath)

This sample document can afford the reader similar features as described in the http://idpf.org/epub/renditions/region-nav document.

<smil>
    <head>
        <sync:media type="application/zip+epub" id="comic" src="https://mycomics.com/superduperhero/episode1.epub" />
        
        <sync:media type="audio/mp4" id="comic-sfx" src="https://mycomics.com/superduperhero/episode1-sfx.mp4" sync:volume="0.7" />
        <sync:media type="audio/mp4" id="comic-sfx-blam" sync:volume="0.7" clipBegin="233s" clipEnd="235s" src="https://mycomics.com/superduperhero/episode1-sfx.mp4" />
        <sync:media type="audio/mp4" id="comic-sfx-wham" sync:volume="0.7" clipBegin="236s" clipEnd="238s" src="https://mycomics.com/superduperhero/episode1-sfx.mp4" />

        <sync:media type="audio/mp4" id="comic-score" src="https://mycomics.com/superduperhero/episode1-score.mp4" sync:volume="0.9" />
        <sync:media type="audio/mp4" id="comic-score-cover" idref="comic-score" clipBegin="1.5s" clipEnd="55s" repeatCount="indefinite" />
        
        <sync:media type="audio/mp4" id="comic-narration" src="https://mycomics.com/superduperhero/episode1-narration.mp4" />
    </head>
    <body type="comicbook">
        <par id="cover" type="cover">
            <audio idref="comic-score-cover" sync:volume="0.7" />
            <seq id="cover-main-panel" type="panel">
                <par id="coverpanel-masthead">
                    <audio idref="comic-narration" clipBegin="10s" clipEnd="15s" />
                    <sync:mediaref idref="myComic" sync:selector="epubcfi(/6/4[cover]!/)" panZoom="150, 100, 400, 120" />
                </par>
                <par id="coverpanel-illustration">
                    <audio idref="comic-narration" clipBegin="16s" clipEnd="21s" />                   
                    <sync:mediaref idref="myComic" 
                        sync:selector="epubcfi(/6/4[cover]!/4)" 
                        sync:clipPath="M 49,58 H 146 C 154,58 160,64 160,72 V 114 C 160,121 154,128 146,128 H 49 C 41,128 3,121 35,114 V 72 C 35,64 41,58 49,58 Z"
                        panZoom="150, 100, 400, 120" />
                </par>
            </seq>
        </par>
        <par id="page1">
            <sync:mediaref idref="soundtrack" fill="repeat" />
            <seq>            
                <seq id="page1-panel1" type="panel">
                    <par id="page1-panel1-bubble1" type="bubble">
                        <audio idref="comic-narration" clipBegin="10s" clipEnd="15s" />
                        <sync:mediaref idref="myComic" 
                            sync:selector="epubcfi(/6/4[page1]!/)" 
                            sync:clipPath="M 43,65 H 189 V 157 H 43 Z"
                            panZoom="20, 20, 175, 120" />
                    </par>
                    <par id="page1-panel1-bubble1" type="bubble">
                        <audio idref="comic-narration" clipBegin="10s" clipEnd="15s" />
                        <sync:mediaref idref="myComic" 
                            sync:selector="epubcfi(/6/4[page1]!/)" 
                            sync:clipPath="M 43,65 H 189 V 157 H 43 Z"
                            panZoom="20, 20, 175, 120" />
                    </par>
                </seq>
                <seq id="page1-panel2" type="panel">
                    <par id="page1-panel2-bubble1" type="bubble">
                        <audio idref="comic-narration" clipBegin="10s" clipEnd="15s" />
                        <sync:mediaref idref="myComic" 
                            sync:selector="epubcfi(/6/4[page1]!/)" 
                            sync:clipPath="M 7,13 H 152 V 179 H 7 Z"
                            panZoom="20, 120, 80, 120" />
                    </par>
                    <par id="page1-panel2-bubble2" type="bubble">
                        <audio idref="comic-narration" clipBegin="10s" clipEnd="15s" />
                        <sync:mediaref idref="myComic" 
                            sync:selector="epubcfi(/6/4[page1]!/)" 
                            sync:clipPath="M 6,227 H 203 V 290 H 6 Z"
                            panZoom="20, 120, 175, 120" />
                    </par>
                </seq>
            </seq>
        </par>
    <body>
</smil>

OCF PACKAGE EXAMPLE

Files are represented as cursive

  • OCF

    • META-INF

      • container.xml
      • metadata.xml
      • encryption.xml
      • etc

    • my-sync-comic.smil

    • assets

      • sounds
      • text
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
    <rootfiles>
        <rootfile full-path="my-sync-comic.smil"
            media-type="application/smil+xml" />
    </rootfiles>
</container>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment