Coding 101 45 (Transcript)

Father Robert Ballecer: Today on Coding 101 its Santa’s Little Helper, part 2.

Netcasts you love, from people you trust. This is TWIT! Bandwidth for Coding 101 Is provided by CacheFly. At cachefly.com.

This episode of Coding 101 is brought to you by Lynda.com. Lynda.com is an easy and affordable way to help you learn. Instantly stream thousands of courses created by experts on business, software, web development, graphic design, and more. For a free trail, visit Lynda.com/c101. Lynda.com/c101. And by Squarespace. The all in one platform that makes it fast and easy to create your own professional website or online portfolio. Now introducing SquareSpace 7 with even better site management tools and other improvements. Try the new Squarespace at squarespace.com and enter offer code C101 at checkout to get 10% off.

Fr. Robert: Welcome to Coding 101, it’s the TWIT show where we let you into the wonderful world of the code monkey. I’m Robert Ballecer the digital Jesuit, and joining us today, our code warrior, Mr. Lou Maresca. Lou, so good to have you back on Coding 101.

Lou Maresca: Thanks for having me.

Fr. Robert: We did skip last week because it was Thanksgiving, did you have a good Thanksgiving?

Lou: I was in Ohio for Thanksgiving, it was great. Little cold, little snowy. Not too bad.

Fr. Robert: Last time we were on we started up a program which chat room called Santa’s Little Helper. The whole idea was to create an application that would automatically check prices of items that you wanted to watch over the holiday season. Let’s say you wanted to get someone a big screen TV and you wanted to watch Newegg and Amazon and whatever it might be, you gave us the framework to do that.

Lou: Yep. It’s just the basic start of being able to call different APIs based off of the URL you pasted in. so if you went to the site, found the item you wanted, grabbed the URL, pasted it in there and then hit enter, and boom you got the details behind the pricing and then you can continue to run that over and over again and then there will be code later on to save all the different historical prices for that item.

Fr. Robert: Right. Of course we made that code available. So if you went to the show notes you could download an entire working package, but we’re going to be evolving that program over the next few episodes so that at the end you should be able to have a program that you can customize exactly to your liking and get your own Santa’s Little Helper. But Lou, before we do that do you think we could talk about some of the ongoing things in the world of programming?

Lou: Let’s do it.

Fr. Robert: This is interesting. We all know that computers are really good at brute force calculations. In fact, back in 1997 when Deep Blue beat Gerry Casperove in Chess, that’s basically what it was. It was the ability to recall just hundreds and thousands of moves every microsecond so it could look at what were the possible outcomes and choose the one that was best for it. It took a while for IBM to program Deep Blue to be able to perform even that feat. But now there are a group of researchers who are looking at a different sort of problem. Not just brute force, but something that requires cataloguing skills. Now Lou, why is cataloguing something difficult for computers?

Lou: The first thing is, there has to be digamma around the data that you have. And the computer doesn’t necessarily know exactly what you’re talking about unless it has the ability to sift through that data. For instance, if I talked about ducks and then I talked about hockey, it doesn’t understand that there is a ducks and a hockey team called Ducks, and it doesn’t understand if I’m talking about an animal or a sport. So when you categorize the data, it kind of narrows or filters the search data down to a specific set and it’s easier for it then to iterate over that smaller set.

Fr. Robert: Right, and that’s again, something that computers are exceptionally good at once they have the category. So if we give a computer 12 different categories, and we give it a huge data set, it will be able to sort that data set into those categories much faster than a human would ever be able to. But as you said, making that cognitive leap has always been a challenge for computer science. For example I wanted my computer to look through data pertaining to the space program. And I have one article that talks about rockets and I have another that talks about liquid oxygen. My brain is going to put those together. Because I understand that liquid oxygen is necessary for the production of rockets. However, a computer, unless it had a predetermined template that said I should put those two together, would not understand that those are related articles. This has been the basic problem with building around fuzzy logic. In other words, trying to imitate how the human brain works. Not just making categories and making correlations, but also going across fields in order to build up a database in our minds. So that’s something that humans have been better at than computers forever. Except maybe not now. There’s an article here about how a team at the University of Wisconsin has developed a machine that can actually at least match, but many times out duel humans, in the extraction and cataloguing of data in the large scale. It was a team of researchers at the U of Madison led by Christopher Reid, a professor at the university. They created a computer that can extract data from scientific publications and place it in a database with tens of thousands of other studies. Again, this has been difficult because computers don’t think like humans. They don’t make those natural correlations. But they created a system that can do that. They specifically created for the paleo-biology database. It’s a huge system that the NSF and other international organizations dump their studies into. They’ve been able to create a software system that looks through the studies, make the correlations and offer them up for a database of paleo-biology. Lou, what would you have to do to make that step? Of course this is all cutting edge, but as a computer programmer, what are the things that you would look at doing in order to let a computer have that power of recognition?

Lou: The first thing is, it’s very similar to how search engines work today. They go out and they web crawl all the websites, they look at links, data on those links, and then they kind of index that data. So that would be the first thing I would need to do is go out and all the data that they were actually researching into or needed as research, I would have to go and crawl it, index it, and find a way to break it down into categories and actual physical useful data to be able to correlate it. So that would be the first thing.

Fr. Robert: Yeah, that indexing is the first part to having the ability to create a database that can be used by big data. Big data is all the rage. We’ve been talking about it for the last 18 months, maybe even 2 years. It’s the idea of pulling in multiple disparate data sources in order to make correlations. That’s good and all but if you don’t have data in the database to pull from, then big data doesn’t work. What the team is hoping is that now that they’ve created this correlative system that can look through studies and pull out those indexed pieces, that they can run it though a big data engine and give the world a paleo-database that anyone can make big data correlations from. It’s kind of cool. Now Lou, when we come back I want to go ahead and jump right back into Santa’s Little Helper. We want to do a review so people remember where we were at the end of the last episode, but before we do that can we take a break because I’d like to talk about our first sponsor of Coding 1o1, that’s Lynda. What is Lynda.com? Lynda.com is a one stop shop. A repository for knowledge. Both’ of new knowledge and knowledge that you just need a refresher course on. Lynda.com is an easy and affordable way to help you learn. You can instantly stream thousands of courses created by experts on software, web development, graphic design, and more. Lynda.com works directly with industry experts and software companies to provide timely training, often the same day you get the new releases on the new versions on the street. You’ll find new courses on Lynda. So you’re always up to speed. All courses are produced at the highest quality. Which means it’s not going to be like a YouTube video with shaky video or bad lighting or bad audio. They take all that away because they don’t want you to focus on the production, they want you to focus on the knowledge. They include tools like searchable transcripts, playlists and certificates of course completion, which you can publish to your LinkedIn profile. Which is great if you’re a professional in the field and you want your future employers to know what you’re doing. Whether you’re a beginner or advanced, Lynda has courses for all experience levels, which means they’re going to be able to give you that reference that place to go back to when you get stumped by one of our assignments. You can learn while you’re on the go with the Lynda.com apps for iOS and Android and they’ve got classes for all experience levels. One low monthly price of $25 gives you unlimited access to over 100,000 video tutorials, plus premium plan members can download project files and practice along with the instructor. If you’ve got an annual plan, you can download the courses to watch offline. Making it the ultimate source of information. Whether you’re completely new to coding or you want to learn a new programming language, or just sharpen your development skills, Lynda.com is the perfect place to go. They’ve got you covered. They’ve got new programming courses right now including the Programming the Internet of Things with iOS, Building a Note taking app for iOS 8, and Building Android and iOS apps with Dreamweaver CC and Phone Gap. For any software you rely on, Lynda.com can help you stay current with all software updates and learn the ins and outs to be more efficient and productive. Right now we’ve got a special offer for all of you to access the courses free for 10 days. Visit Lynda.com/c101 to try Lynda.com free for 10 days. That’s Lynda.com/c101. Lynda.com/c101. And we thank Lynda for their support of Coding 101. Let’s get back to the action. Lou, in the last episode, we showed people how to access basic data through APIs. However, you also showed them how you can access the APIs of companies who don’t necessarily want you to have access to their APIs. Can you tell us a little bit about what your thought process was through that?

Lou: Yes, like you said, a lot of the retailers like Best Buy or Amazon give you the ability to search through their data and actually pull out based off of key words or based off of a skew to identify their items. But some retailers are not into that. Allowing 3^rd parties to build applications to pull data from their database. And maybe it’s because they don’t have the scale in their hardware to be able to handle that. Maybe they can handle a bunch of web users but not a bunch of API calls. And so they don’t have that ability. Maybe it’s just they didn’t want to make it public. So for instance, Newegg is one of them. They have what they call private API where they have special partners that can kind of sign up and use their API to pull data about parts and items on their site but they don’t have a public one. So one of the things you’d have to do is kind of reverse their site in a way where you can use that private API to kind of pull the data that you need without having to do what we call screen scrape from their website. So that’s kind of what I do for one of them. But Best Buy and some of the other ones, they give you this API and then they just kind of throttle you based on how many calls you make, but it’s pretty easy to use their API.

Fr. Robert: Right.  What we did was we broke down the app we wanted to create into that fourfold category that we introduced ten weeks ago. The first part was to figure out what we want the functions of the app to be. In this case we wanted the app to be able to look for certain items on websites. On certain services. Maybe compare prices. But essentially just access information about an item from multiple sources. The second thing that we suggested doing is to gather resources. In other words, what do I have available for me. And what Lou did was show us that we have all these APIs. Either public or hidden APIs that we can access so that we don’t have to do screen scraping. And Lou did talk a little bit about that in the last episode. Lou, want to refresh their memories on what screen scraping means?

Lou: Sure. It’s just what we call our “technical term” for pulling data that’s unstructured from websites. So if you go to a website today and you look at the underlying HTML of that website, there is a bunch of data that I can click on the price of an item on the webpage and then there’s a little tag underneath that has the price element. And so what I could do is write some special code that goes in and looks through the HTML of every webpage and then extracts that price out and saves it off. And so it’s kind of a very fragile piece of code because anytime a site might change their website or might change some of their elements, their naming conventions, maybe the way they set their prices, then that code might break. So you have to basically go in and fix that code. But screen scraping is just an ability, if a site doesn’t have an API to call, you can just go to the webpage and pull the data out that way.

Fr. Robert: Right. And someone in the chatroom is saying well, is that legal? It sounds like a funny question but it is a very good question. Because you’ve got cases, like for example, ticket master sued to stop several screen scraping apps from taking their prices and putting them into their application. What ticket master was saying was that’s proprietary information, we’re showing it to the public, but its copyright, you can’t just take it. And what those other companies where saying was that if you put it on the internet, and it’s just text, you can’t copy write just text that randomly goes up on the web. They’ve gone back and forth and I don’t actually know where they left that case, but that is something to consider. If you are dealing with information from a company that really doesn’t want to make the information public, you may be running into some legal, ethical, moral, grey area. So just know that. Be a little careful.

Lou: That’s really a slippery slope if you think about it. Because that’s really what search engines do. They go to every webpage in the entire internet space and they pull data out of them. So if you were to go to Google and search for Ticket Master price, I’m sure that they’ll probably point you to Ticket Master. So I think that’s a slippery slope. But either way, you should be very careful when you’re using nonpublic type APIs.

Fr. Robert: Right. I’m not saying don’t do it, I’m just saying if you do do it, you’re probably not going to find a lot of sympathy when it breaks because eventually they’re just going to change one thing on the page and you’ll have to rewrite your screen scraping app. we say that all the time. Now the third part was to build a logic tree. In other words, once we know what we want the app to do, what resources we have available to do it, to figure out what will be the components of our application. So do we need to bring something in that we have to write a function to bring it in? Do we need something else to be able to compare values that we have to write a function to do that? Do we need to write another class that’s going to be responsible for pushing that data back out to the user in some understandable format that we need to make sure that there’s a logic branch for that? And the last part is to make each part of the logic tree a reality. And that’s pretty much where we are now. Last week Lou, you gave us working code. Actual code that people could download to get this up and running. We’re going to do a little something different today. Tell me in broad strokes, what do you want our audience to learn by the end of today’s episode?

Lou: I think what we want to do is go over calling one of these APIs and making sense of the data that we get back. And then be able to use it inside your code. So I think we kind of said we built off the structure, we gave you the concepts last time, we said okay, this is how you do it, how you break it down. But let’s actually show you how to do it and then this way you can kind of use that to apply it to maybe other retailers that we don’t actually give you. So I think that might be the best to kind of teach you how to fish and then let you go fish, so to speak.

Fr. Robert: Give a man a fish and he’ll shop for a day, teach him how to use the APIs and he’ll shop forever? I don’t know, that doesn’t work. That’s a fantastic plan. So that’s where we’re going to be going right after this break. I want to do one more ad so that we can bring you home uninterrupted, but essentially, we’re going to build on the foundation we had from last week. Now remember, you’re all going to get every last line of code that Lou is going to be showing you. It’ll be in the show notes, you can download it immediately, you’ll be able to open it up, run it, play with the code, and Lou, am I right in assuming you want people to try this code with other web presence right? Who are you giving this code for?

Lou: I think anybody who wants to, right now we’re going to give you a couple, we have Walmart, and we have Best Buy. Feel free to kind of extend that and add your own whatever you need. There’s a lot of public APIs we’re not taking into account here, expand it, see what you can do with it.

Fr. Robert: Have you tried Amazon?

Lou: You do have to sign up to get an API key so they can track you, what you’re actually looking at. But they do have what they call a private API too which is what their app uses. It’s really all dependent on what you want to use and if you want to be able to track your stats too. So I normally use a public one but it doesn’t matter which one you use.

Fr. Robert: Fantastic. Speaking of using public keys and using resources that are available to us, what if you want to make your resources available to others? You could go the traditional route, you could rent a host and you get your domain and make sure that you get the one that’s exactly right for you and you could do all your own programming. Install PHP and make sure that you’re using a framework like Word Press, but altered just a tiny bit so it better suits your- yeah. You could do all that or you could try SquareSpace. Squarespace is that one place you can go to on the internet to easily get your project or portfolio on the internet and I love Square Space because they’re always improving their platform. In fact they just released Squarespace 7. That makes getting started much easier and they have a unique web presence that is built up over templates that they give you. It’s now more all in one, it’s simpler to navigate and it’s simpler to operate in 1 seamless experience. Squarespace 7 allows you easier editing. You can edit on 1 screen which means you no longer have to toggle between site manager and preview mode. You can even preview designs in divide modes - that you see exactly how it will look on tablets and mobiles. Now it also offers instant access to professional stock photography from Getty. It’s now integrated into the package. They’ll allow direct purchases inside the platform from getting images at just $10 each for your site. No more having to jump out, get an image imported into Squarespace, make sure all your licensing is taken care of. It’s all in 1 interface. You can get google branded email with Squarespace 7 so you can have the branded email for your small business and it’s automatically set up when you set up your Squarespace account. They’ve got templates, designed for specific professions which this is a big one. All their templates have been beautiful but now they give you the option to switch between different classes of templates. You can move between templates for musicians, artists, architects, chefs. They designed those templates, those category specific templates so that you can find the one that looks right for the project, for the business, for the industry that you’re trying to represent. On Squarespace 7 the developer platform is now on a beta which means that you can customize your site exactly as you wish. If you’re a developer you have access to the same platform that Squarespace uses for its own site – complete code control They also give you e-commerce with all subscription plan levels that includes the ability to accept donations which is great for non-profits, cash wedding registries and school fund drives and it’s easy to use. Yes, sometimes you’ll run into a ram but it’s easy to get it solved because Squarespace offers you support; 24 hours a day 7 days a week. They’ve got an army of folk in their forums giving you self-help articles and video workshops to browse at your leisure. It also starts at just $8 a month so it’s not going to break the bank. That includes a free domain if you sign up for a year. The Squarespace portfolio, the note, metric and blog mobile apps are on the go extensions of your website so that you can monitor and make changes from anywhere. They include the hosting so again it becomes a 1 stop shop. You don’t have to worry about buying different services from different providers. It’s all from 1 place. It’s all in a square space. Here’s what we want you to do. We want you to start a free 2 week trial with no credit card required and start building your website today. When you decide to sign up for Squarespace make sure to use the offer code c101 to get 10% off and to show your support for Coding 101. To begin using Squarespace 7 now existing customers can go to the setting tab to activate all of your new features. We thank Squarespace for their support of Coding 101. A better web awaits and it starts with your new Squarespace website. Lou, show me the money. I’ve got my basic understanding of what it looks like when I use a JSON API so I can get information back or when I screen scrape, but you want to show us specifically how to access Best Buy.

Lou: That’s right. So I think really quick we’ll go over a couple things. We’ll go over calling the API with a keyword, and this is like for instance, if I wanted to search for Xbox 1, I want to call the API with a key word, then I get this JSON blob back. And this is a big piece of text that basically sections off the data that comes back about the item. And then how can I convert that into code so that I can actually read and query out let’s say the price of an item skew or whatever? And then they extract that price out. So that’s kind of what I want to do in the code today.

Fr. Robert: Actually Lou, it’s interesting because when you talk about getting that JSON, we showed that in the last episode, what a JSON blob actually looks like. And it really is a blob. It’s everything, just flooding back into your computer. It’s interesting that you have that step in there, to convert it into strong types, because we need to go from the blob to something that we can actually use. In fact, that’s a lot like the banter item that we talked about at the top of the show, where you can have all the information, but unless you know how to categorize it, you won’t actually be able to use it.

Lou: Exactly. So I think one of the things that Best Buy does well is they actually have a really simple API. Actually, I might not be the best fan of Best Buy sometimes, but they have a really good API, it’s very simple to use. And one of the things that I’ve been trying to do is strong type it in a way that I don’t have to do too much work to get it done. So for instance, right now they have a rest API. So what I want to do is I want to show how you can build that query, that API call, very easily, without having to do much work. And then when they get the data back I actually create strong types for it by using a utility that will convert the JSON to C Sharp code. And so it makes it not even take two minutes here. So first, I sent the code to you guys that give you kind of like the framework where we have a base provider called a price provider. And in there it has 3 separate methods that you can get it by using a URL, or you can get it by using an ID. But what I want to do is add to this and say that in Best Buy’s case, I might want to query based off of a key word and say they give me this item and then the price for that item. So that’s a little different then what this was- so I’m actually going to add a method to Best Buy one. Where it’s what we call return by keyword. So I’m going to add a method here called return by keyword. And so what this’ll allow me to do is go get by keyword. But the thing is, I need to be able to call the API so I’m going to add in here what we call a search query. I’m going to add a new class, called the Best Buy search query. And what this is going to do is I’ve already given you a base class in here already, called a URL and coding parts- and what this does is if we look at the Best Buy API, they give you this special URL that you need to call. And if you look in here there’s a question mark and then there’s a bunch of data after that. And that’s what we call a query string. So when you call a rest API you can put a bunch of query strings. Of course you add your API key in here and then the name of the item and some other things. So what we want to do is go and create a query sting for being able to search by keyword. So I’ve actually added a special utility class in here called the URL and code parser that makes it really easy, I’ll show you how easy it is. So what I’ll need to do is, ill actually inherit from that. And ill inherit the URL parser and what this will allow me to do now is I can just build up a parse for this URL. So really easily ill pop in here and I’ll just paste the class. And you’ll see here that I have a search query and it inherits some parser and now I’ve added the parameters that I require. The first thing is the base URL. It’s a special search list URL. And then there’s the ST parameter after that question mark and that’s the search text. And then the number of pages I want and the number of rows and then of course what I want to sort by. And I can do sort by bestselling or most frequent or that sort of thing. So this is the very simple use of an API to basically be able to query by keyword.

Fr. Robert: What this is doing is that first line is just a standard URL. You can hit that from anywhere in the world, it will call up a very specific location on that server, it will call the function searchlist.jsp and then everything below it is you’re adding items to your query. Your query string you showed us earlier when you were looking at the Best Buy API. You’re just saying put this in the query, this in the query, this in the query, and build it up until I’m done and then drop it at the end of that URL.

Lou: That’s right. So it makes it really easy. So once I pop back over to this code I can write some simple code here that will allow me to kind of query that using that using that new query provider. So I think the first thing here is my code wants to return the strong type that I called price search result. If we look at what that is, again it’s just a framework class that I created that has some information about the API and that’s kind of agnostic of the retailer. So for instance, the title of it, the price, and its ID, that kind of thing. So I want to return that so that’s the first thing I need to do. I want to return my product even though it doesn’t have any data in it. So then the code kind of goes in between here. So then I want to build up my query. So the first thing I want to do is- and the nice thing is I immediately have the search query thing that I imported my class that I created.  And in there I want to set the keyword which is what I pass in here. And then I want to sort by best match. And that’s really all I have to do and I’m done. So now what I want to do is create the URL for it. And the way you do that is you call the two string method. And if we look in here, the two string method basically just grabs all of the query parameters that I put in there like the ST and the sort, and it just pins them to the end of the URL. And so the URL that I get is the one that’s very similar to the one on the Best Buy site that I showed you. So that’s it. Now I need to be able to call the API and return the data.

Fr. Robert: This is for the Best Buy API but most APIs are going to run very close to this. They may change the queries that you can make, but they all work on a URL that you pin some sort of query to, yes?

Lou: Exactly right, yeah. Most of them are rest APIs. Some of them are what we call post or a package. And that means that you have to put it in the body of the HTTP request. And so that’s a little more complicated. Amazon does that sometimes. But more are rest API. Most are just the URL with a bunch of query parameters on the end of it that kind of ask for a specific type of data.

Fr. Robert: Right. And if you want to know the different ways of submitting information to a server, I believe we covered that in module 2. We specifically talked about what the pros and cons of doing it each way, personally I kind of like putting everything in the URL. It’s nice and neat to me. But some people do like having them where you have to have it in the body because you can do certain things with it.

Lou: Exactly right. Really quick let’s test the API. So I’m going to go and add a new project, I’m just going to do a console app right now called test. And then in here I’m going to call the Best Buy price provider and it should ask me do you want to add that class provider. I go in here and add the solution that I just created which is this whole class down here. And then it says oh yeah, you have a Best Buy priority created, so I’ll go add that using statement, boom, there. And then all I need to do is create a test. Ill return the price and that’s from calling the keyword that I added, which is this guy. And here I’ll put Xbox 1. So this is our first test. So if we go and run this…

Fr. Robert: Now you’re hardcoding this, but if we were actually making this app for use, we would never want to hard code a specific product.

Lou: That’s right. This is just for a test for us to first check out if the code is working correctly or not. So I’m going to set this project as the default one so that it will run. This is just a console app here but I’m going to move that out of the way. Let’s step through it. So the first thing that we’re going to see is we build up our search query and then we’re going to create a HTTP client that allows us to make the call. And here’s where we make the call out there. So the key here is, what will end up happening is, that made the call so if we kind of hover over this, inside here, you’ll see this massive blob of JSON. And it’s kind of hard to see in here but it’s kind of a mess.  So what we can do is actually copy this and I’m going to show you a really cool tool that I found online. It’s called JSON Utils.

Fr. Robert: For anyone watching at home, that’s what came back. So after he submits that URL with that query, that’s why we call it a blob. It literally looks like a blob of stuff. But there is information in there that we want.

Lou: Right. If you looking here you can kind of see some things I need. Like I need this skew for that’s the Best Buy unique identifier, so I need to pull that out, be able to pull the price. It has the hardware ID and all that. So I’m going to copy this out and use a special utility called the JSON Utils. And this Util, what it will do, is you can basically paste a blob in here and you can convert it to a bunch of classes down here, which is really cool. So I’ll just go ahead and submit that and now it’s actually created a bunch of Best Buy classes.

Fr. Robert: No way, that is beautiful. Where is this?

Lou: This is called JSON Utils.

Fr. Robert: Okay, we’ve got to give them the link for that. Seriously. If you were to type in that query, if you were just to hand type that query, you would get back that blob on your screen. It would show up in your browser. So what this will do, it’ll actually take that blob and turn it into useful categories. Otherwise you’d have to program it by hand. This gorgeous.

Lou: It’s brilliant. So I thought it was a fantastic tool. Basically what I’m going to do is add that class now, I’m going to add a special class, Best Buy Search Results. And then in here I’m going to paste the code that I got back from there. So I’ll go over to the site and I’m going to grab all these classes that they created in there. Copy and paste it in there. And then add a reference there. And now I’m done. So now if you can see in here is, if you go up to the very top, the one that I really want to pay attention to is this item document class. What it’ll return is the skew ID and some information that I need about the item. So if I close this, now I can actually add back in here the code that I put in here and there’s a special thing that says hey, use the search results I just got back from that utility, and convert it to the actual strong- type version of it. And what I can do then is I can go and grab the first search result out and extract the skew. So there we go. And then I already have a class that I gave you that will take the skew, the ID and convert it to the price. Or the actual product. So that’s what this line will do right here.

Fr. Robert: The JSON de-blob-ifyer already put all the data into strong type. So if you type in the skew it should already have a product associated with it.

Lou: Exactly right. So if we run this now, let’s just run it and stop it right there. And if we step through it, you notice we get this blob back again, there’s this massive blob here. And then now the next line should convert into a strong type. So now if I hover over it I’ve got a search result. It’s got 20 documents in it and here’s all the 20 different return types and the first one being the Xbox Unity console, Assassin’s Creed, and sure enough, if I scroll down one of the properties is called skew ID. And that’s really what I want. There it is. Fantastically easy to do when you convert it to strong type.

Fr. Robert: When you do it this way it looks ridiculously easy. This is not a whole lot of code, this is essentially using what someone else has already done, a couple of lines to pull the data in. someone else’s utility to turn the blob back into useable information, and then it’s just comparison. You’re just doing comparisons.

Lou: Best Buy does have fantastic documentation on the stuff that comes back. For instance, understanding the document and that there’s many documents and then inside that document, the properties of that document. They do explain that on their API site, so check that out but it’s fairly self-explanatory, once you strongly type it and you look through the code. You start looking through what you get back. You kind of understand, okay well I’ve got a bunch of documents, there’s 20 search results, because I said I wanted the minimum amount of list names I wanted to come back was 20. And then I want to pick the one out that I actually need to use to get the skew ID and that would be the first document. If you look back at the code here, I’m going to pull the skew ID and then I’m going to pass it back into my GIT price by skew, and then the next thing is now I’ve got my product and there’s my price. And so pretty easy to do. That’s really just using some of the utilities that you have out there and of course, the API that Best Buy provides, so pretty simple stuff.

Fr. Robert: Now Lou, I’ve got to ask, because this is great, very cool. But it’s one manufacture. It’s one retailer. The challenge is when you start to say okay, I want this Santa’s Little Helper to actually compare prices and as we know, most manufactures aren’t going to use the same skews. They’re not going to use the same identifying information, maybe not even the same title of the product. So you do still, if you’re going to do multiple vendors, it’s not just as easy as downloading all the JSON blobs from the different manufactures, and then comparing skews, because those might not be the same. What do you have to do to match those up?

Lou: So one of the commonalities between sites, especially Amazon, Amazon uses what they call ASI and numbers, Best Buy uses skew IDs, New Egg uses their egg IDs or whatever. They all have different unique identifiers for their retail items in their stock, but what they do use that’s common is what they call model numbers or model IDs. So for instance, if I’m looking for a specific model of a TV, there’s a special model ID that usually Samsung will give. So the key here is, if you’re going to be linking the data between all the different sites, you have to use something common. So that’s really something very unique and very common among all the sites. If I search by a model number in the keyword here, I’m going to get back that item and that’s going to be the data that I need to use. So that’s kind of the key is, using this specific unique identifiers across them. If you look inside the base class I provided, it’s called price provider base class, retail product. You’ll see here I have the ID and the retail unique ID. So the retail unique ID is explicit to that, for example Walmart or the ASN from Amazon, or the specific product from Sam’s Club or whatever. But then the ID itself would probably be the model number to basically be the same across all products. So that’s all you need to do is to find a unique identifier that will allow you to go search amongst all the other retail sites.

Fr. Robert: Actually this is one of the reasons, I actually asked a programmer about this, one of the reasons why if you go to different retail stores, sometimes they have the exact same product, but the last two letters or numbers have changed to denote that it’s a product that’s only available at that retail store, and a lot of times its to prevent people from scraping their prices. Because if they’re just looking at the bits of data in the JSON blob, they’re not going to find anything that correlates with the product from another company or retailer or site. That’s something that as you make Santa’s Little Helper more advanced, you can actually hard code those differences, because retailers have no obligation to make sure that you can directly compare just from the JSON blob.

Lou: If you’re building an application with an UI, what you could potentially do is present it to the user. So you could go search Best Buy- I think that there’s a site out there that does this for prices for airplanes too, where you go and it’ll search all the different websites for planes and it’ll say okay, here’s Priceline, and here’s this one over here and they kind of give you side by side comparisons. So in the UI you could potentially say okay, here’s Walmart’s results, here’s Best Buy’s, here’s whatever, choose the right item. Then you’ll go in and say yeah, this is the same item here and checkbox it or something. And then that’s the one that will actually save off. So you kind of give more power to the user rather than making the application do all the work, you kind of let the user decide which item is the same.

Fr. Robert: It’s funny that we’re doing this because this is actually what a search on Google or Bing will do if you’re looking for a product. It will tell you, you can find it here, here, here, and here. They’re using a similar function to go through these different retailers, scrape the data off so they can make it searchable on their own search engines. So hey, you’re just doing one that’s targeted for you. Lou Maresca, I want to thank you so very much for being part of this episode of Coding 101, it is always such a joy to have you on. I love that JSON Parser. I’m going to play with it right after the episode. Could you tell the folks what we’re going to be doing next week?

Lou: You bet. So the first thing is, we’re going to wrap a UI around it, like we promised from the beginning. As well as we want to be able to store that data around all the different prices we get over a long period of time. So let’s say you run it once a day for an entire week, you want to store those prices, be able to query them, and be able to present that in a UI, we can kind of see the trends. So that’s where we can slop a UI around it and maybe even a database to store that data in and that’ll kind of exit us out to an application you can actually use.

Fr. Robert: I can’t wait. It’s nice to have an application that people are actually going to be able to use. Even if you don’t want to learn how to program this, the application that you’re going to be able to get from us actually will be useful. Lou can you tell the folks at home where they can find you? We all know that you’re a senior product developer, a software engineer for Microsoft. Working on some wonderful CRM products. But where else can they find the work that you do?

Lou: You bet. Anything posted on twitter, LouMM, and of course the aboutme.loumm. Check that out. I’ve been posting some new stuff I’ve been working on outside of work here. And that should show up there as well. And then of course all the work I do here at Microsoft is at crm.dynamics.com.

Fr. Robert: Lou Maresca, our code warrior. Thank you so very much. We offer a prayer of thanksgiving for your code warrior skills.

Lou: Thank you very much. Take care.

Fr. Robert: That’s all the time we have for this episode of Coding 101. Don’t forget to go to our show page. Twit.tv/code. You can find all the episodes of Coding 101. Also don’t forget you can find us on YouTube. Youtube.com/twitcoding101, if that’s the method of watching our content that you fancy. On our show page there is a download link if you want to get automatically subscribed on your iPhone, iPad, mac, PC, whatever way you want it. Don’t forget our Google+ group. Plus.google.com/twitcoding101, you can find out what’s going on with our community. This is a great place to ask questions if you’re having trouble or if you have a project to take to the next level. You can find me at twitter.com/padresj. If you follow me on twitter you’ll find out what I’m doing on every episode of Coding 101 as well as all the other shows I do here on the twit TV network. Come watch us, come follow, be part of the TWiT army. We do this show live every Thursday at 1:30 pm. Drop by at live.twit.tv. You can watch the pre-show, post-show and all the bloopers we take out of the final mix. As long as you’re watching live, jump into the chatroom at irc.twit.tv so you can talk to us. Thank you for everyone who makes this show possible. Lisa and Leo, for letting us do it. And to my super TD, Mister Brian, Cranky Hippo himself. Where can they find you on the TWiT TV network?

Bryan Burnett: Me? Earlier on Thursdays with you doing Know How. Also tune in and watch All about Android on Tuesdays. You can also follow me on twitter @Cranky_Hippo. Most recently I posted a corgi butt for @corgibuttsj.

Fr. Robert: Thank you. I feel the love. Until next time. I’m father Robert Ballecer, this has been Coding 101.  End of line!