{ "items": [ { "type": [ "http://schema.org/BlogPosting" ], "properties": { "accessMode": [ "textual" ], "accessModeSufficient": [ "textual" ], "articleBody": [ "\nWhile YouTube is free (as in money) to use, the cost is paid in terms of privacy and advertising analytics. So I've decided to investigate self-hosting my video content.\n\n\nThe Cost of YouTube\nWith YouTube, you sacrifice privacy in favor of cost. YouTube is the very best at what they do (serve video to all resolutions and bandwidths), and they are backed by Google who is the very best at what they do (collect data in order to facilitate selling a primed audience to advertisers).\n\n \n \n \n \n\n\nThere’s nothing inherently wrong with that. We live in a capitalistic society; there is money to be made; Google/YouTube is providing a service to advertisers; many consumers will (knowingly or unknowingly) give up their privacy in exchange for free-as-in-money services.\nBut as I’ve gotten older and started to realize just how much data Google has on each and every one of us, I’ve started valuing my privacy a lot more. I’d like to provide an option for you to protect your privacy as well.\nSelf-Hosting Video Content\nEven with efficient video codecs, video can still cost a lot of money to serve.\nMany websites provide a video to their users, wherein this video is a single file, and the browser will begin loading and playing the video from start to finish. This means that even if the user only watches the first few seconds of a 5 minute video, it’s possible that the video is downloaded in its entirety — which is an unnecessary cost.\nHowever, we can provide a better user experience as well as reduce hosting costs by leveraging the ability to serve bandwidth-adaptive chunks of video to players on-demand.\nAdaptive Bitrate Streaming\nThere are two major, semi-compatible approaches to adaptive bitrate streaming over HTTP. One is called HTTP Live Streaming (“HLS”), and the other is called Dynamic Adaptive Streaming over HTTP (“MPEG-DASH”).\n\n \n \n \n \n\n\nFrom Wikipedia:\n\nAdaptive bitrate streaming is a technique used in streaming multimedia over computer networks. While in the past most video or audio streaming technologies utilized streaming protocols such as RTP with RTSP, today’s adaptive streaming technologies are almost exclusively based on HTTP and designed to work efficiently over large distributed HTTP networks such as the Internet.\nIt works by detecting a user’s bandwidth and CPU capacity in real time and adjusting the quality of the media stream accordingly. It requires the use of an encoder which can encode a single source media (video or audio) at multiple bit rates. The player client switches between streaming the different encodings depending on available resources. “The result: very little buffering, fast start time and a good experience for both high-end and low-end connections.” […]\nHTTP-based adaptive bitrate streaming technologies yield additional benefits over traditional server-driven adaptive bitrate streaming. First, since the streaming technology is built on top of HTTP, contrary to RTP-based adaptive streaming, the packets have no difficulties traversing firewall and NAT devices. Second, since HTTP streaming is purely client-driven, all adaptation logic resides at the client. This reduces the requirement of persistent connections between server and client application. Furthermore, the server is not required to maintain session state information on each client, increasing scalability. Finally, existing HTTP delivery infrastructure, such as HTTP caches and servers can be seamlessly adopted.\nA scalable CDN is used to deliver media streaming to an Internet audience. The CDN receives the stream from the source at its Origin server, then replicates it to many or all of its Edge cache servers. The end-user requests the stream and is redirected to the “closest” Edge server. […] The use of HTTP-based adaptive streaming allows the Edge server to run a simple HTTP server software, whose licence cost is cheap or free, reducing software licensing cost, compared to costly media server licences (e.g. Adobe Flash Media Streaming Server). The CDN cost for HTTP streaming media is then similar to HTTP web caching CDN cost.\n\nThis means that we can use off-the-shelf services like Amazon S3 and Amazon CloudFront to serve video, which are relatively inexpensive and have large user-bases who can answer questions when you run into issues.\nHTTP Live Streaming (HLS)\nAfter doing some research, I came across a blog post that was particularly helpful — “Self-hosted videos with HLS” by Vincent Bernat.\nVincent writes:\n\nTo serve HLS videos, you need three kinds of files:\n\nthe media segments (encoded with different bitrates/resolutions),\na media playlist for each variant, listing the media segments, and\na master playlist, listing the media playlists.\n\nMedia segments can come in two formats:\n\nMPEG-2 Transport Streams (TS), or\nFragmented MP4.\n\nFragmented MP4 media segments are supported since iOS 10. They are a bit more efficient and can be reused to serve the same content as MPEG-DASH (only the playlists are different). Also, they can be served from the same file with range requests. However, if you want to target older versions of iOS, you need to stick with MPEG-2 TS.\n\nAt the time of this writing, iOS 12 will be out in a week or two. A quick search tells me that iOS 10 and newer make up 85% of all iOS users. This means that I can pretty safely use the Fragmented MP4 method which, according to these sources, is more compatible with MPEG-DASH for some cross-over implementations in the future.\nSample Video\n\n \n \n \n \nSource: Hallelujah - Brooklyn Duo (Piano + Cello)\nImplementation\nEncoding and Deploying Video\nVincent Bernat provides a tool on GitHub which greatly simplifies the process of creating the various video fragments called video2hls.\nFor this website, I have put together a workflow for creating and serving HLS video content.\n\n\nI use H.264 video with AAC audio wrapped inside an MP4 container, exclusively. These are all defined as part of the MPEG-4 specification, and is the best-supported grouping of codecs and containers across all browsers and devices.\nHardware-level decoders are commonplace inside computers, phones, tablets, and set-top boxes like Xbox, PlayStation, and Apple TV.\n\n\nI have a directory called streaming-video, which is separate from the images that I use and push to S3. Video files are large, and I don’t want to accidentally push partially-completed video data to my caching CDN before they’re ready.\n\n\nI have a command which takes any video file inside the streaming-video folder, with a filename ending in -source.mp4, and passes it through video2hls, creating a folder called {video}.fmp4 which contains all of the video and playlist files I need across a large variety of bandwidths and resolutions.\nIt will only do the work to create the directory and all of the fragmented files if the directory doesn’t already exist.\nfind ./streaming-video -type f -name \"*-source.mp4\" | xargs -I {} \\\n bash -c 'if [ ! -d \"${1%-source.mp4}.fmp4\" ]; then \\\n video2hls --debug --output \"${1%-source.mp4}.fmp4\" --hls-type fmp4 \"$1\"; \\\n fi;' _ {} \\;\n\n\nI find all of the .m3u8 playlist files and gzip them (since they’re just text). This is essentially an in-place rewrite of the files.\nfind ./streaming-video -type f -name \"*.m3u8\" | xargs -P 8 -I {} \\\n bash -c '! gunzip -t $1 2\u003e/dev/null \u0026\u0026 gzip -v $1 \u0026\u0026 mv -v $1.gz $1;' _ {} \\;\n\n\nLastly, I push all of the files up to the hls folder in my S3 bucket using the AWS Unified CLI Tools, setting the correct Content-Type and Content-Encoding headers.\n# The .m3u8 playlists that we gzipped\naws s3 sync ./streaming-video s3://blog.ryanparman.com/hls \\\n --exclude '*.*' \\\n --include '*.m3u8' \\\n --acl=public-read \\\n --cache-control max-age=31536000,public \\\n --content-type 'application/vnd.apple.mpegurl' \\\n --content-encoding 'gzip'\n\n# The video \"posters\"\naws s3 sync ./streaming-video s3://blog.ryanparman.com/hls \\\n --exclude '*.*' \\\n --include '*.jpg' \\\n --acl=public-read \\\n --cache-control max-age=31536000,public \\\n --content-type 'image/jpeg'\n\n# The fragmented MP4 files\naws s3 sync ./streaming-video s3://blog.ryanparman.com/hls \\\n --exclude '*.*' \\\n --include '*.mp4' \\\n --acl=public-read \\\n --cache-control max-age=31536000,public \\\n --content-type 'video/mp4'\n\n\nThe Client-Side Code\nAfter pushing the content to our CDN, we can use the standard HTML5 \u003cvideo\u003e tag to tell browsers how to load the requested assets.\n\u003cvideo poster=\"https://cdn.ryanparman.com/hls/hallelujah.fmp4/poster.jpg\" controls preload=\"none\"\u003e\n \u003csource src=\"https://cdn.ryanparman.com/hls/hallelujah.fmp4/index.m3u8\" type=\"application/vnd.apple.mpegurl\"\u003e\n \u003csource src=\"https://cdn.ryanparman.com/hls/hallelujah.fmp4/progressive.mp4\" type='video/mp4; codecs=\"avc1.4d401f, mp4a.40.2\"'\u003e\n\u003c/video\u003e\n\n\nHere, we have a static poster image that the \u003cvideo\u003e element loads by default.\n\n\nNext, we have an HLS-compatible playlist file (.m3u8), which ultimately points to the correct .mp4 files.\n\n\nLastly, we have a standard .mp4 fallback.\n\n\nEnabling Chrome, Firefox, and Edge using hls.js\nDailymotion has released a JavaScript library called hls.js which enables HLS playback on browsers like Chrome, Firefox, and Edge using Fragmented MP4 sources.\nYou can load the script from the CDN:\n\u003cscript src=\"https://cdn.jsdelivr.net/npm/hls.js@latest\"\u003e\u003c/script\u003e\nAfter that, we have the implementation. Here, we start with a working \u003cvideo\u003e element, then use JavaScript to swap over to HLS.\n(() =\u003e {\n 'use strict';\n\n if (Hls.isSupported()) {\n let selector = \"video source[type='application/vnd.apple.mpegurl']\",\n videoSources = document.querySelectorAll(selector);\n\n videoSources.forEach(videoSource =\u003e {\n let m3u8 = videoSource.src,\n once = false;\n\n // Clone the video to remove any source\n let oldVideo = videoSource.parentNode,\n newVideo = oldVideo.cloneNode(false);\n\n // Replace video tag with our clone.\n oldVideo.parentNode.replaceChild(newVideo, oldVideo);\n\n // On play, initialize hls.js, once.\n newVideo.addEventListener('play', () =\u003e {\n if (once) {\n return;\n };\n once = true;\n\n var hls = new Hls({\n capLevelToPlayerSize: false\n });\n hls.attachMedia(newVideo);\n hls.loadSource(m3u8);\n hls.on(Hls.Events.MANIFEST_PARSED, (event, data) =\u003e {\n newVideo.play();\n });\n }, false);\n });\n }\n})();\nCORS\nIf you are serving the files from a third-party host (such as Amazon S3), you will need to enable CORS support on your bucket.\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cCORSConfiguration xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"\u003e\n \u003cCORSRule\u003e\n \u003cAllowedHeader\u003e*\u003c/AllowedHeader\u003e\n \u003cAllowedOrigin\u003e*\u003c/AllowedOrigin\u003e\n \u003cAllowedMethod\u003eGET\u003c/AllowedMethod\u003e\n \u003cAllowedMethod\u003eHEAD\u003c/AllowedMethod\u003e\n \u003c/CORSRule\u003e\n\u003c/CORSConfiguration\u003e\nAdditionally, if you have a CDN cache in front of that S3 bucket (e.g., Amazon CloudFront), you’ll need to make sure that it is configured to allow the Origin headers through and also respond to the HTTP OPTIONS verb.\nYou can find more information about solving this problem with CloudFront at “Configuring CloudFront to Respect CORS Settings”.\n\n\n\n" ], "author": [ { "type": [ "http://schema.org/Person" ], "properties": { "alumniOf": [ { "type": [ "http://schema.org/Organization" ], "properties": { "name": [ "https://wepay.com", "WePay" ], "url": [ "https://wepay.com" ] } }, { "type": [ "http://schema.org/Organization" ], "properties": { "name": [ "https://aws.amazon.com", "Amazon Web Services" ], "url": [ "https://aws.amazon.com" ] } } ], "familyName": [ "Parman" ], "gender": [ "male" ], "givenName": [ "Ryan" ], "honorificPrefix": [ "Mr." ], "image": [ "https://cdn.ryanparman.com/hugo/gravatars/current/8c73b5fc88b88ac0d92fec541c6a4413890df291.jpg" ], "jobTitle": [ "engineering manager" ], "knowsAbout": [ "software development", "site reliability engineering", "security", "CI/CD", "SRE" ], "knowsLanguage": [ "en" ], "memberOf": [ { "type": [ "http://schema.org/WebSite" ], "properties": { "name": [ "Flailing Wildly" ], "url": [ "https://ryanparman.com" ] } } ], "name": [ "Ryan Parman" ], "nationality": [ "United States of America" ], "sameAs": [ "https://codepen.io/skyzyx/", "https://coderwall.com/skyzyx", "https://hub.docker.com/u/skyzyx/", "https://github.com/skyzyx", "https://libraries.io/github/skyzyx", "https://www.npmjs.com/~skyzyx", "https://pypi.org/user/skyzyx/", "https://forums.aws.amazon.com/profile.jspa?userID=62398", "https://dev.to/skyzyx", "https://news.ycombinator.com/user?id=skyzyx", "https://www.reddit.com/user/skyzyx", "https://stackoverflow.com/users/228514/ryan-parman", "https://cash.me/$rparman", "https://www.paypal.me/rparman", "https://venmo.com/skyzyx", "https://rparman.yelp.com/", "https://itunes.apple.com/profile/skyzyx", "https://bandcamp.com/skyzyx", "https://www.last.fm/user/skyzyx", "https://pandora.com/profile/ryan4516", "https://soundcloud.com/skyzyx", "https://open.spotify.com/user/skyzyx", "https://www.airbnb.com/users/show/47877510", "https://amazon.com/wishlist/KAFYR57E8R81", "https://browser.geekbench.com/user/skyzyx", "https://homescreen.me/skyzyx", "https://www.kickstarter.com/profile/skyzyx", "https://unsplash.com/@skyzyx", "https://angel.co/skyzyx", "https://betalist.com/@skyzyx", "https://www.linkedin.com/in/rparman", "https://feedly.com/skyzyx", "https://www.goodreads.com/user/show/7693108-ryan-parman", "https://medium.com/@skyzyx", "https://hackerone.com/skyzyx", "https://keybase.io/skyzyx", "https://twitter.com/skyzyx", "https://imdb.com/user/ur24445966/", "https://letterboxd.com/skyzyx/", "https://trakt.tv/users/skyzyx", "https://www.youtube.com/user/Skyzyx", "https://aws.amazon.com/sdk-for-php/", "https://www.mheducation.com", "https://github.com/simplepie", "https://wepay.com" ], "worksFor": [ { "type": [ "http://schema.org/Organization" ], "properties": { "name": [ "https://www.mheducation.com", "McGraw-Hill Education" ], "url": [ "https://www.mheducation.com" ] } } ] }, "id": "https://ryanparman.com/posts/2018/serving-bandwidth-friendly-video-with-hls/#author" } ], "copyrightHolder": [ { "type": [ "http://schema.org/Person" ], "properties": { "alumniOf": [ { "type": [ "http://schema.org/Organization" ], "properties": { "name": [ "https://wepay.com", "WePay" ], "url": [ "https://wepay.com" ] } }, { "type": [ "http://schema.org/Organization" ], "properties": { "name": [ "https://aws.amazon.com", "Amazon Web Services" ], "url": [ "https://aws.amazon.com" ] } } ], "familyName": [ "Parman" ], "gender": [ "male" ], "givenName": [ "Ryan" ], "honorificPrefix": [ "Mr." ], "image": [ "https://cdn.ryanparman.com/hugo/gravatars/current/8c73b5fc88b88ac0d92fec541c6a4413890df291.jpg" ], "jobTitle": [ "engineering manager" ], "knowsAbout": [ "software development", "site reliability engineering", "security", "CI/CD", "SRE" ], "knowsLanguage": [ "en" ], "memberOf": [ { "type": [ "http://schema.org/WebSite" ], "properties": { "name": [ "Flailing Wildly" ], "url": [ "https://ryanparman.com" ] } } ], "name": [ "Ryan Parman" ], "nationality": [ "United States of America" ], "sameAs": [ "https://codepen.io/skyzyx/", "https://coderwall.com/skyzyx", "https://hub.docker.com/u/skyzyx/", "https://github.com/skyzyx", "https://libraries.io/github/skyzyx", "https://www.npmjs.com/~skyzyx", "https://pypi.org/user/skyzyx/", "https://forums.aws.amazon.com/profile.jspa?userID=62398", "https://dev.to/skyzyx", "https://news.ycombinator.com/user?id=skyzyx", "https://www.reddit.com/user/skyzyx", "https://stackoverflow.com/users/228514/ryan-parman", "https://cash.me/$rparman", "https://www.paypal.me/rparman", "https://venmo.com/skyzyx", "https://rparman.yelp.com/", "https://itunes.apple.com/profile/skyzyx", "https://bandcamp.com/skyzyx", "https://www.last.fm/user/skyzyx", "https://pandora.com/profile/ryan4516", "https://soundcloud.com/skyzyx", "https://open.spotify.com/user/skyzyx", "https://www.airbnb.com/users/show/47877510", "https://amazon.com/wishlist/KAFYR57E8R81", "https://browser.geekbench.com/user/skyzyx", "https://homescreen.me/skyzyx", "https://www.kickstarter.com/profile/skyzyx", "https://unsplash.com/@skyzyx", "https://angel.co/skyzyx", "https://betalist.com/@skyzyx", "https://www.linkedin.com/in/rparman", "https://feedly.com/skyzyx", "https://www.goodreads.com/user/show/7693108-ryan-parman", "https://medium.com/@skyzyx", "https://hackerone.com/skyzyx", "https://keybase.io/skyzyx", "https://twitter.com/skyzyx", "https://imdb.com/user/ur24445966/", "https://letterboxd.com/skyzyx/", "https://trakt.tv/users/skyzyx", "https://www.youtube.com/user/Skyzyx", "https://aws.amazon.com/sdk-for-php/", "https://www.mheducation.com", "https://github.com/simplepie", "https://wepay.com" ], "worksFor": [ { "type": [ "http://schema.org/Organization" ], "properties": { "name": [ "https://www.mheducation.com", "McGraw-Hill Education" ], "url": [ "https://www.mheducation.com" ] } } ] }, "id": "https://ryanparman.com/posts/2018/serving-bandwidth-friendly-video-with-hls/#author" } ], "copyrightYear": [ "2018" ], "dateModified": [ "2018-09-09T03:18:33+00:00" ], "datePublished": [ "2018-09-09T03:18:33+00:00", "2018-09-09T03:18:33Z" ], "description": [ "While YouTube is free (as in money) to use, the cost is paid in terms of privacy and advertising analytics. So I've decided to investigate self-hosting my video content. The Cost of YouTube With YouTube, you sacrifice privacy in favor of cost. YouTube is the very best at what they do (serve video to all resolutions and bandwidths), and they are backed by Google who is the very best at what they do (collect data in order to facilitate selling a primed audience to advertisers).", "\nWhile YouTube is free (as in money) to use, the cost is paid in terms of privacy and advertising analytics. So I've decided to investigate self-hosting my video content.\n" ], "headline": [ "Serving Bandwidth-Friendly Video with HTTP Live Streaming (HLS)" ], "image": [ "https://cdn.ryanparman.com/hugo/posts/2018/adaptive-bitrate-streaming.png", "https://cdn.ryanparman.com/hugo/posts/2018/youtube-2017.png", "https://cdn.ryanparman.com/hugo/home/default-oembed.jpg" ], "keywords": [ "youtube,h.264,hls,mpeg,dash,streaming,s3,cloudfront,safari,chrome,firefox,edge," ], "mainEntityOfPage": [ { "type": [ "http://schema.org/WebPage" ], "properties": {} } ], "name": [ "Serving Bandwidth-Friendly Video with HTTP Live Streaming (HLS)" ], "publisher": [ { "type": [ "http://schema.org/Organization" ], "properties": { "logo": [ { "type": [ "https://schema.org/ImageObject" ], "properties": { "height": [ "60" ], "url": [ "https://cdn.ryanparman.com/hugo/favicons/apple-touch-icon-60x60.png" ], "width": [ "60" ] } } ], "name": [ "Flailing Wildly" ] } }, { "type": [ "http://schema.org/Organization" ], "properties": { "logo": [ { "type": [ "https://schema.org/ImageObject" ], "properties": { "contentUrl": [ "https://cdn.ryanparman.com/hugo/favicons/apple-touch-icon-60x60.png" ], "height": [ "60" ], "url": [ "https://cdn.ryanparman.com/hugo/favicons/apple-touch-icon-60x60.png" ], "width": [ "60" ] } } ], "name": [ "Flailing Wildly" ] }, "id": "https://ryanparman.com/posts/2018/serving-bandwidth-friendly-video-with-hls/#publisher" } ], "wordCount": [ "1611" ] } }, { "type": [ "https://schema.org/BreadcrumbList" ], "properties": { "itemListElement": [ { "type": [ "https://schema.org/ListItem" ], "properties": { "item": [ "https://ryanparman.com" ], "name": [ "HOME" ], "position": [ "1" ] } }, { "type": [ "https://schema.org/ListItem" ], "properties": { "item": [ "https://ryanparman.com/posts/" ], "name": [ "POSTS" ], "position": [ "2" ] } }, { "type": [ "https://schema.org/ListItem" ], "properties": { "item": [ "https://ryanparman.com/posts/2018" ], "name": [ "2018" ], "position": [ "3" ] } } ] } }, { "type": [ "http://schema.org/FollowAction" ], "properties": { "url": [ "https://twitter.com/skyzyx" ] } } ] }