Parser API using .Net Core Web API 2.1 and Hosting in Raspberry PI

Posted on Edited on Disqus:

This article explains how to create generic parser API to parse video feeds from various sites that supports RSS/Atom feed or scraping the content from DOM element and return the custom XML / JSON output based on requested streaming box supported format. The core parsing logic will be implemented in base class and then custom formatting logic will be implemented in the appropriate child classes. Later in the article, we will also deploy this application into Raspian based Raspberry PI. We are using the latest .Net Core SDk 2.1 RC release for developing the web API. In my earlier article , i developed the similar application using NodeJS and Express and deployed in Raspberry PI.

Design

The below diagram depicts the architecture of Model and Service Components. In this article, i have added only for feed based service and i will be adding to support DOM based extraction later and publish it into github.

Model Class Diagram

image

IParserModel - Interface for the model.

public interface IParserModel
{
    string RawContent { get; set; }       
}

BaseParserModel – Base class for the Parser Mode. RawContent will holds the Raw content of feed or DOM.

public class FeedBaseParserModel : BaseParserModel
    {
        [XmlIgnore]
        public List<isyndicationitem> SyndicationItems { get; set; }
    }

RokuParserModel – This is the root class for Roku which holds all the properties and subclasses that are expected for Roku Streaming Box. I will be adding AndroidParserModel and IOSParserModel in future to support other streaming boxes. The RokuParserModel also have sub classes to hold other properties such video URL and ThumbnailURL. This can be customized in whatever way our streaming box expects the model to be.

[XmlRoot(ElementName ="Feed")]
    public class RokuFeedParserModel : FeedBaseParserModel
    {        
        [XmlElement]
        public int ResultLength { get; set; }
        [XmlElement]
        public int EndIndex { get; set; }      
        [XmlElement(ElementName ="Item")]
        public List<rokuparseritem> ParserItems { get; set; }
    }
 
    [XmlRoot(ElementName = "Item")]
    public class RokuParserItem
    {
        [XmlElement]
        public string Title { get; set; }
        [XmlElement]
        public int ContentId { get; set; }
        [XmlElement]
        public string StreamFormat { get; set; }
        [XmlElement]
        public string Synopsis { get; set; }
        [XmlAttribute]
        public string SdImg { get; set; }
        [XmlAttribute]
        public string HdImg { get; set; }
        [XmlAttribute]
        public string ThumbnailURL { get; set; }
        [XmlElement]
        public RokuMediaItem MediaItem { get; set; }
    }
 
    [XmlRoot(ElementName = "MediaItem")]
    public class RokuMediaItem
    {
        [XmlElement]
        public string StreamUrl { get; set; }
        [XmlElement]
        public string ThumbnailURL { get; set; }
    }

Service Architecture

image

IParserService – Base Interface for Service

public interface IParserService<t> where T : IParserModel
    {
        Task<t> ParseContent();
    }

BaseParserSerivce - Abstract Layer for Paser Service Class. This will have empty virtual method for now.

public abstract class BaseParserService<t> : IParserService<t> where T : IParserModel, new()
    {       
        public async virtual Task<t> ParseContent()
        {
            return await Task.FromResult(new T());
        }
    }

RokuFeedParserService – This is the service class for the Roku Format that holds all the core logic extract the content items from the feed. It takes the feedURL in constructor and override the parseContent method to implement Feed Based Parsing Service. I used the SyndicationFeed library from .net to parse RSS and atom feed. The base RokuFeedParserService will parse the feed and populate the list of items in SyndicationItems Property. Later, the child class that will be inhertiing from RokuFeedParserService will use the SyndicationItems values to create custom formatted the xml / json output based on the streaming format requested.

public class RokuFeedParserService : BaseParserService<rokufeedparsermodel>
    {
        public string FeedURL { get; set; }
 
        public RokuFeedParserService(string _feedURL)
        {
            FeedURL = _feedURL;
        }
 
        public async override Task<rokufeedparsermodel> ParseContent()
        {
            RokuFeedParserModel parserModel = new RokuFeedParserModel() { SyndicationItems = new List<isyndicationitem>() };
            using (XmlReader xmlReader = XmlReader.Create(FeedURL, new XmlReaderSettings() { Async = true }))
            {
                var reader = new RssFeedReader(xmlReader);
                while (await reader.Read())
                {
                    switch (reader.ElementType)
                    {
                        case SyndicationElementType.Item:
                            parserModel.SyndicationItems.Add(await reader.ReadItem());
                            break;
                    }
                }
            }
            return parserModel;
        }
    }

Ch9RokuParserService – This is the child service class for Channel9 feed and it will override the parseContent method to populate RokuFeedParserModel object based on SyndicateItems values. This will return the final output of Roku based parser model object. We will be adding additional service classes for supporting other formats here.

public class Ch9RokuParserService : RokuFeedParserService
   {
       
       public Ch9RokuParserService(string _feedURL) : base(_feedURL)
       {
           
       }

       public async override Task<rokufeedparsermodel> ParseContent()
       {
           var parserModel = await base.ParseContent();
           parserModel.ParserItems = new List<rokuparseritem>();
           int currIndex = 0;
           foreach(var syndicationItem in parserModel.SyndicationItems)
           {
               RokuParserItem parserItem = new RokuParserItem();
               parserItem.Title = syndicationItem.Title;
               parserItem.ContentId = currIndex;
               parserItem.StreamFormat = "mp4";
               parserItem.MediaItem = new RokuMediaItem();
               parserItem.MediaItem.StreamUrl = syndicationItem.Links.FirstOrDefault(i => i.RelationshipType == "enclosure")?.Uri.ToString();
               parserModel.ParserItems.Add(parserItem);
               currIndex++;
           }
           parserModel.ResultLength = currIndex;
           parserModel.EndIndex = currIndex;
           return parserModel;
       }

   }

Controllers

BaseAPIController – This is the base API controller and it will have the default annotationattributes for APIController and Route Actions. Note that i am using APIController Attribute that denotes a Web API controller class and it provides some useful methods and properties by coupling with ControllerBase method such as automatic 400 responses and more. I have also defined the default route [Route(“api/[controller]“)] at base class level so that i dont have to redefine this on every other controller.

[ApiController]
    [Route("api/[controller]")]
    public class BaseAPIController : ControllerBase
    {        
        public BaseAPIController()
        {
 
        }
    }

Ch9Controller – This controller will have all the GET methods for various streaming boxes and produces the xml or json output. Note that, i am using HttpGet(“Roku”) so that it allows me to have multiple GET methods on single controller. you can also define your Routing by action like [HttpGet(“[action]”)] and then you can call the API with the method name like /API/Ch9/GetRokuFormat

public class Ch9Controller : BaseAPIController
    {       
        [HttpGet("Roku")] 
        [Produces("application/xml")]
        public RokuFeedParserModel GetRokuFormat()
        {           
            var parserService = new Ch9RokuParserService("https://s.ch9.ms/Feeds/RSS");            
            return parserService.ParseContent().Result;   
        }
    }

Startup.cs - In my startup class, i have enabled both XML and JSON formatter to support both format based on the request. You can also create custom formatter
if you need other than XML / Json. Note that i have enabled RespectBrowserAcceptHeader = true to support the XML output. Also, i have used the XML Annotation to change the
element name and added the XMLIgnore Attribute to ignore from serialization. 

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }
 
    public IConfiguration Configuration { get; }
 
    // This method gets called by the runtime. Use this method to add services to the container.
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc(options =>
        {
            options.RespectBrowserAcceptHeader = true;                
        })
        //support application/xml
        .AddXmlSerializerFormatters()
        //support application/json
        .AddJsonOptions(options =>
        {
            // Force Camel Case to JSON
            options.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();
        });
    }
 
    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }
 
        app.UseMvcWithDefaultRoute();
    }
}

Program.cs - I used the default settings to start the kestrel web server and configured to the use the 5000 port for hosted application to listening on that port. This will be used when we deploy the application Raspberry PI later.

public class Program
    {
        public static void Main(string[] args)
        {
            BuildWebHost(args).Run();
        }
 
        public static IWebHost BuildWebHost(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
                .UseStartup<startup>()
                .UseUrls("http://*:5000")
                .Build();
    }

Now that we have done the coding for our channel9 feed Web API to support Roku XML format, let us run the application first using IIS express to make sure it works.

image

Deploying .Net Core Web API on Raspberry PI

Now, that our application produced the xml output as expected, we will deploy this latest .Net Core 2.1 Web API in our Raspberry PI. Please note that .Net Core runs only on Raspberry PI 2 / 3. It does not run on Pi Zero. The Raspberry PI that i have is currently running on Raspian OS.

Before we deploy our application, As a first step, we have to install the .Net Core SDK and Runtime on Raspberry PI. In order to install the SDK, we will be executing the below commands on PI terminal window. I already have remote connection enabled from my laptop for my PI. I have also enabled network share from my PI so that i can publish the code later using windows file share. If you want to know how to enable to remote connection and file sharing for your Raspberry PI , visit the daveJ article about Beginner’s Guide to Installing Node.js on a Raspberry Pi and he explained all the steps in details.
Launch the remote connection and connect to the PI Server. Launch the terminal window and run the following commands.

$ sudo apt-get -y update
$ sudo apt-get -y install libunwind8 gettext
$ wget https://dotnetcli.blob.core.windows.net/dotnet/Sdk/2.1.300-rc1-008673/dotnet-sdk-2.1.300-rc1-008673-linux-arm.tar.gz
$ wget https://dotnetcli.blob.core.windows.net/dotnet/aspnetcore/Runtime/2.1.0-rc1-final/aspnetcore-runtime-2.1.0-rc1-final-linux-arm.tar.gz
$ sudo mkdir /opt/dotnet
$ sudo tar -xvf dotnet-sdk-2.1.300-rc1-008673-linux-arm.tar.gz -C /opt/dotnet/
$ sudo tar -xvf aspnetcore-runtime-2.1.0-rc1-final-linux-arm.tar.gz -C /opt/dotnet
$ sudo ln -s /opt/dotnet/dotnet /usr/local/bin

The first two commands are required for Raspbian for deploying .Net Core SDK and Runtime. These are some dependencies modules that has to be added manually. For more details, you can check the official documentation here.

The next two WGET commands will download the latest DotNet SDK and Runtime (2.1 RC1) and then the following commands will be used to extract the output to /opt/dotnet folder and a symbolic link created for dotnet.

If all the above steps are done with no errors, .Net Core SDK is installed on PI. Just run the command dotnet –info to display the information about DotNet SDK and Runtime details.

Now, that we have installed the .Net Core SDK and Runtime on PI, its time to build and deploy the published code on PI. As I mentioned earlier, I have the network shared drive enabled on my PI to copy the files. You can also transfer to PI via other methods like FTP.

As a first step, Lets publish the application in Linux ARM architecture since Raspbian is based on Linux. Navigate to the project folder and execute the following command to publish the output.

dotnet publish . -r linux-arm

image

if you want to publish in release mode, you can add -c release attribute.

The code is published in Linux-arm\Publish folder. Now, I will create a folder in PI server called VideoParserAPI and copy all the files from linux-arm\publish folder and paste into VideoParserAPI folder in PI Server.

Now the code is published in to PI, we will just the run the application to start the service listening on port 5000. Remember in my startup class, i used the Port No 5000 to listen for network connection. Remember, this is my personal project and i will be using it only in my internal network and i have no intention to publish it on internet. If you have the application that needs to be published on internet, you may have to use reverse proxy like nginx to configure port 80 and reroute to kestrel web server for best practices.

image

Lets run the application to start the services. Open the Terminal Window on PI Server and execute the ./VideoParserAPI to run the application. It a few seconds, the service will start and listening on Port 5000.
image

Lets call the web API from my system to see the output.

http://pi:5000/api/ch9/roku

image

There you go. Web API Application developed in Latest .Net Core 2.1 is running on Raspberry PI. Smile

Conclusion

I have created design pattern to handle all the core parsing logic in the base classes so that we don’t have to rewrite the logic for every other streaming boxes. however, we can customize the logic to change the output xml / json content based on the streaming box in appropriate child class.

In future, I will be developing an android app to consume these web api to play the content in my mobile with chromecast support. We can also extend this library to any other streaming box (Apple TV, Fire TV) . I also planned to deploy this web API app inside the docker in my Raspberry PI later.

I hope this will help you to get going with your crazy ideas.

The entire source code is uploaded to github.

Happy Coding !!!