Interesting project, Tom! I should plan more before coding.

Is Code Planner free or paid?

It's free :)

Already tried that. Not worth it with the current token limitations.

But I'm currently tinkering with other applications of AI and MrScraper. Will ship something when I think it's reliable enough.

I'm experimenting with something like you described, but there are lots of edge cases and needs a bit of polishing. But for now, I made a Chrome extension that helps tinkering with CSS selectors.

The onboarding process is not a fillable form, it's just a simplification of the scraper builder to show how it works. You just have to click next.

Other users have also reported that this is a bit confusing, so I'm going to start working on improving this.

Thanks for you feedback!

If the target page or DOM changes, it's not considered a failed scraping, just the data you wanted will return empty.

But you gave me the idea to send an alert or email notification if a scraper stops returning data or content changes.

Thanks

totally, that's going to be a huge piece of functionality that will add a ton of value to the user. It's what I always worry about in maintaining my scrapers, updating xpaths, all the rigmarole parsing and what not. Stable pages are much appreciated but we all know web designers are gonna design. Thanks for making this product, being in a post https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn world is gonna make for some great adventures I think. Keep up the good work.

I could depending on the proxy I got from my provider. I'm currently working on adding the ability to select a higher quality proxy for difficult to scrape websites and to add captcha solvers as well.

But if I have to be honest, I can not guarantee it at the present time.

This app is the side-project I starter 2 months ago, it's evolving fast but I still need to add some key features for enterprise customers.

Honestly I haven't tried Octoparse, but doing a quick check, I can see it is way more expensive.

Sorry for not being more clear. The onboarding process is a simplified version of the actual scraper builder. The fields are not editable, it's just to get used to the scraping flow.

I've noted down your suggestion and I'll make this more clear or add field edition.

Thanks!

Yeah, this took me a minute to figure out, too. I'd change it to make it clear those are just static slides. Even better, remove it entirely, and when the user lands on the home page (after verifying), open a wizard that guides them through setting up their first scraper.

I also had a real problem naming the "Store as" field for my data extractor. It didn't seem to like things in the format "foo_bar_baz56" (i.e. ending with digits). This page https://mrscraper.freshdesk.com/support/solutions/articles/1... says "The variable name can not contain special signs" but doesn't explain what special signs are. Anything other than [A-Z_]?

Now that I've finally set up and tested my first scraper, I'm really impressed. It was much easier to set up than I would have guessed, and specifying a selector made it dead simple. Results worked out of the box, on a site that is super touch about being scraped.

However, now that I'm viewing my scraper, I see no way of editing the scraper or data extractor. What's the trick to editing a scraper once you've saved it and gone back to view it?

Thanks for your precious feedback!

I've noted everything down and I'll improve the onboarding experience, fix and make variable names more understandable and improve the edit button.

A scraper is not editable once is queued to run or currently running, then, you need to reload the page for the edit button to appear again. I will improve this.

Thank you again

Thanks for this amazing feedback. I can get some action items from this.

I've noted down your comment and I'll be improving things for next week!

Thank you so much. Appreciate it!