Logging into a website and gathering data

A place to discuss the implementation and style of computer programs.

Moderators: phlip, Moderators General, Prelates

User avatar
Turtle_
Posts: 186
Joined: Mon Apr 14, 2008 10:27 pm UTC

Logging into a website and gathering data

Postby Turtle_ » Mon Jan 03, 2011 4:11 am UTC

My friend is in IB HL Computer Science, and for one of the IB requirements he needs to make some sort of application. Our school uses this website, Edline, to post student grades. My friend's program is going to take the grade data from the website, then display it additional useful ways. I told him I would write the part for gathering data, which is fine for IB as long as he references that he was using someone else's code for part.

I started learning about how to interface with websites in Java (which is what we're most familiar with, and what he'll be using) and I've managed to download regular files, such as .html or .txt, from websites using Java's URLConnection class. The issue is that the program would have to log into the website, which I'm not sure how to do. I'm not even sure if Java is the best way to do this, and I'm open to learning something else if there's a much easier way to do this.

A picture of https://www.edline.net/Index.page homepage (you can go and look at the source) and one of the class pages
Spoiler:
Main page.png

screenshot2.png

The grades are in a simple html table so they're easy to read, I just need help getting the html file. Note that the grades pages have multiple frames.

In short, I need help with making a program that logs into a website and does a bit of navigation.
"Sometimes lies were more dependable than the truth." ~ Ender's Game
"Ignorance more frequently begets confidence than does knowledge." ~ Charles Darwin

keeperofdakeys
Posts: 658
Joined: Wed Oct 01, 2008 6:04 am UTC

Re: Logging into a website and gathering data

Postby keeperofdakeys » Mon Jan 03, 2011 4:47 am UTC

Get your web browser, disable javascript and then try logging in. If you can, then this means that you should be able to login without too much effort.

User avatar
headprogrammingczar
Posts: 3072
Joined: Mon Oct 22, 2007 5:28 pm UTC
Location: Beaming you up

Re: Logging into a website and gathering data

Postby headprogrammingczar » Mon Jan 03, 2011 1:17 pm UTC

Do that, then read the HTTP spec. It will help immensely.
<quintopia> You're not crazy. you're the goddamn headprogrammingspock!
<Weeks> You're the goddamn headprogrammingspock!
<Cheese> I love you

User avatar
Berengal
Superabacus Mystic of the First Rank
Posts: 2707
Joined: Thu May 24, 2007 5:51 am UTC
Location: Bergen, Norway
Contact:

Re: Logging into a website and gathering data

Postby Berengal » Mon Jan 03, 2011 5:39 pm UTC

Get Apache's httpclient library, then look for something to parse html.

And yes, read the HTTP spec. Try logging into the site manually using something like telnet or netcat.
It is practically impossible to teach good programming to students who are motivated by money: As potential programmers they are mentally mutilated beyond hope of regeneration.

User avatar
Turtle_
Posts: 186
Joined: Mon Apr 14, 2008 10:27 pm UTC

Re: Logging into a website and gathering data

Postby Turtle_ » Tue Jan 04, 2011 1:20 am UTC

You do need javascript to login. I'll read the HTTP spec, then get back in a few days if I need anymore help. Thanks.
"Sometimes lies were more dependable than the truth." ~ Ender's Game
"Ignorance more frequently begets confidence than does knowledge." ~ Charles Darwin

User avatar
Berengal
Superabacus Mystic of the First Rank
Posts: 2707
Joined: Thu May 24, 2007 5:51 am UTC
Location: Bergen, Norway
Contact:

Re: Logging into a website and gathering data

Postby Berengal » Tue Jan 04, 2011 4:00 am UTC

With javascript it becomes a bit harder. Chances are you'll still be able to log in pretty easily by just figuring out what the javascript does. After all, the server doesn't really have any way of verifying that you've run the javascript or not (and it certainly shouldn't care).
It is practically impossible to teach good programming to students who are motivated by money: As potential programmers they are mentally mutilated beyond hope of regeneration.

User avatar
Emu*
Posts: 689
Joined: Mon Apr 28, 2008 9:47 am UTC
Location: Cardiff, UK
Contact:

Re: Logging into a website and gathering data

Postby Emu* » Tue Jan 04, 2011 1:44 pm UTC

Have you considered whether the edline manager would give you a better form of access than page-scraping? A low-privilege SQL account maybe?
Cosmologicon wrote:Emu* implemented a naive east-first strategy and ran it for an hour, producing results that rivaled many sophisticated strategies, visiting 614 cells. For this, Emu* is awarded Best Deterministic Algorithm!

User avatar
Steax
SecondTalon's Goon Squad
Posts: 3038
Joined: Sat Jan 12, 2008 12:18 pm UTC

Re: Logging into a website and gathering data

Postby Steax » Tue Jan 04, 2011 6:00 pm UTC

If you're really desperate, you can just browse in a browser and automate mouse clicks and keystrokes. A really crude method, but eh.
In Minecraft, I use the username Rirez.

User avatar
thoughtfully
Posts: 2253
Joined: Thu Nov 01, 2007 12:25 am UTC
Location: Minneapolis, MN
Contact:

Re: Logging into a website and gathering data

Postby thoughtfully » Tue Jan 04, 2011 7:16 pm UTC

Steax wrote:If you're really desperate, you can just browse in a browser and automate mouse clicks and keystrokes. A really crude method, but eh.

You might find an acceptable compromise by using firefox or chrome extensions.
Image
Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.
-- Antoine de Saint-Exupery

User avatar
Thesh
Made to Fuck Dinosaurs
Posts: 6579
Joined: Tue Jan 12, 2010 1:55 am UTC
Location: Colorado

Re: Logging into a website and gathering data

Postby Thesh » Wed Jan 05, 2011 9:30 am UTC

All you need to do is read and send the cookies, as well as the post data for any forms that it might use. See this for information on how to send/receive cookies:

http://www.hccp.org/java-net-cookie-how-to.html
Summum ius, summa iniuria.

tiki4718
Posts: 1
Joined: Sat Mar 29, 2014 9:14 pm UTC

Re: Logging into a website an' gathering data

Postby tiki4718 » Sat Mar 29, 2014 9:20 pm UTC

Were you able to successfully retrieve the table? I too want to do the same thing


Return to “Coding”

Who is online

Users browsing this forum: No registered users and 9 guests