etl

An interview with Ruby ETL expert Thibaut Barrère

#499 — April 30, 2020

Read on the Web

???? Occasionally we run interviews in Ruby Weekly and we're back with another one.. with long time Rubyist and Kiba maintainer, Thibaut Barrère. Be sure to check out the bottom of this issue to read it, especially if you ever run ETL jobs with Ruby! ????

Ruby Weekly

▶  Let's Build a Twitter Clone in 10 Minutes with Rails, CableReady, and StimulusReflex — You know that cloning Twitter in 10 minutes is impossible, but what about the core mechanism of the idea? After a slow first minute, this video does a pretty good job of showing off some techniques you might not have used before.

Nate Hopkins

discuss.rubyonrails: The Rails Project Discussion Forum — Basically a Web version of the Rails mailing lists and a worthwhile place to head if you want to suggest features, ask questions, etc.

Ruby on Rails Discussions

Easy Rails Deployments — Deploy your Ruby, Rails, Sinatra, and Rack application to any cloud or server. Cloud 66 offers a scalable Heroku alternative that doesn't lock you in. Try it free and get extra $66 free credits with the code: 'Ruby-Weekly'.

Cloud 66 sponsor

sequel-activerecord-adapter: Allows Sequel to Reuse an ActiveRecord Connection — If you want to use more Sequel or migrate in that direction, this makes it easy.

Janko Marohnić

Ruby Adds Experimental Support for 'End-Less' Method Definitions — We’ve touched on this in a previous issue, but if you fancy a proper blog post with examples, this is more accessible than digging through feature tracker discussions. I’m not a fan of this syntax myself yet, but Prateek does a good job of selling it.

Prateek Choudhary

CableReady: Trigger Client-Side DOM Changes from Server-Side Ruby — If you skipped the video above because it’s a video (and I know many of you do ????) CableReady is still worth checking out. It aims to “complete the ActionCable story” by providing a way to directly interact with clients over ActionCable WebSockets. The docs will help you get the idea.

Hopsoft

???? Jobs

Find a Job Through Vettery — Vettery specializes in tech roles and is completely free for job seekers. Create a profile to get started.

Vettery

Security Engineer (Remote) — Are you an engineer with experience in Rails and/or Go? Join our team and help secure our apps and cloud infrastructure.

Shogun

ℹ️ Interested in running a job listing in Ruby Weekly? There's more info here.

???? Articles & Tutorials

A Practical Use for PStorePStore is one of the older parts of Ruby’s stdlib and lets you persist (and restore) a Ruby hash to disk. I haven’t seen it in Ruby code for years but Arkency have found a neat, modern use case.

Paweł Pacana

▶  Drag and Drop Sortable Lists with Rails and Stimulus JS — How to wire up drag and drop lists with a Rails app using Sortable, Stimulus and acts_as_list.

Go Rails

Setting Up Multi-Factor Authentication for RubyGems.org — If you’re a registered user of the official Ruby Gems repository, you should have had an email this week about securing your account using 2FA. This is just my own reminder that this is a good idea especially if you publicly publish widely used gems! :-)

Rubygems.org

Let’s Explore Big-O Notation With Ruby ???? — Learn to use Big-O to look at an algorithm and easily discern its efficiency, without having to run a profiling tool.

Honeybadger sponsor

▶  Discussing Docker and Kubernetes with Kelsey Hightower — A worthwhile show to listen to if Kubernetes and Docker intimidate you but you want to know a little more. Kelsey is good at breaking these things down into understandable pieces.

Rails with Jason Podcast podcast

Rails System Tests in Docker — We’re seeing an uptick in articles about system tests in Rails. Here’s how to integrate them into your development Docker setup.

Hint.io

The Difference Between System Specs and Feature Specs — If you’ve felt the difference between RSpec’s “feature specs” and “system specs” is quite subtle, this explanation will help.

Jason Swett

Why Rubyists Should Consider Learning Go — If you want compilation and a type system, Crystal is probably a better fit for Rubyists, but Go is undoubtedly a neat language and ecosystem (and if you do end up in the Go world, check out our Go weekly! ????)

Ayooluwa Isaiah

Building a Ruby CLI with Thor

Daniel Gómez

???? Code and Tools

git curate: Peruse and Delete git Branches Ergonomically — Got a repo cluttered with branches here and there? git curate aims to cure the pains of getting those branches back under control.

Matt Harvey

MessageBus: A Reliable and Robust Messaging Bus for Ruby and Rack

Sam Saffron

Are You Spending Too Much on Heroku?

Rails Autoscale sponsor

bootstrap_form: A Rails Form Builder for Bootstrap v4-Style Forms

Bootstrap Ruby

ActiveModelAttributes: The Active Record Attributes API, but for Active Model — Brings some of the goodies of the Rails 5 Active Record attributes API to ActiveModel too. 1.6.0 just dropped.

Karol Galanciak

???? A Q&A with…
Thibaut Barrère
Creator of Kiba ETL

Thibaut Barrère is a long-time Rubyist and data engineer who built and maintains the popular Extract, Transform, and Load (ETL) framework Kiba. We asked him some questions about his work:

What inspired you to create Kiba?

A lot of my work since ~2005 has been focused on data integration (making systems speak together), data aggregation etc. I sometimes used GUI-based tools like Microsoft SSIS which, while powerful, are quite far from the coding experience.

I was already using Ruby at that time, and was happy to discover activewarehouse-etl (maintained by Anthony Eden, who runs DNSimple now) providing a Ruby DSL to declare data pipelines. I used it for a while with very good success to implement data extractions and business intelligence ETL, and ultimately took over the maintenance.

In the long run, though (as I explained in a recent Paris.rb talk), the balance between the the cost of OSS maintenance and the usability for my billable and non-billable use-cases proved to be not good enough, which made me decide to stop the maintenance, sadly.

I still wanted to use Ruby to write data pipelines, but I needed to cover more use-cases and reduce the OSS maintenance burden at the same time. This ultimately led me to write and share (in 2015) Kiba ETL, a focused DSL for declarative processing, matching those criterias.

Do you find yourself adding more features while in quarantine?

Before the quarantine, I directed my Kiba bandwith and focus on finalizing Kiba v3 and rewriting the documentation from scratch, to properly encourage best practices I’ve been discovering. I also created experimental branches for Ruby 2.7/2.8 keywords.

During the quarantine, I've reduced client work and OSS work too, to focus on shipping Kiba Pro v2 (which I’ll announce shortly officially). I’ve extracted and generalized (from real-life projects) very useful components, such as a “batch SQL lookup” (useful when replacing relationships keys during data migrations and datawarehouses sync code, in batch rather than row by row), a “file lock” to ensure a single job runs at once, and a “parallel transform” to achieve easy concurrency for things like HTTP queries.

What's the wildest ETL that you've encountered?

Getting the data out of a system which is actively not acting in that direction is always a bit wild.. One can see all types of fancy stuff on the field. For instance, it is not uncommon to have an ETL process start a headless browser, jump through pages, just to get to the CSV/PDF/Excel file that you will then use as your data source!

You can also end up having to figure out ways to read or write very old file formats at times. Recently I wrote a Kiba component to generate a COBOL delimited file, for instance. In large companies, a very widely used I/O is good old SFTP, far away from modern APIs and formats.

Can you tell us how to say your last name? ????

I had to deploy a page to my blog to answer that question properly ????. You’ll find out how to say my name here.

Merci Thibaut!

You can read some of Thibaut's posts on his blog and find out more about Kiba ETL here.




etl

Has Jasleen Matharu secretly married Anup Jalota? The actress clarifies

The current lockdown that will soon complete two months and has already been extended till May 18, has forced all of us to stay safe and inside our houses. There are no gatherings, get-together parties, weddings, or any event. Amid this, is it possible for any celebrity to tie the knot? Fans began to think Bigg Boss fame Jasleen Matharu has, and that too with Anup Jalota.

She took to her Instagram account a few days back and shared a picture where she could be seen with a Sindoor and a pair of bangles that immediately made fans wonder if she had taken the oath with the Bhajan maestro. In case you missed it, have a look at the post here first:

 
 
 
View this post on Instagram

❤️❤️

A post shared by Jasleen Matharu ਜਸਲੀਨ ਮਠਾੜੂ (@jasleenmatharu) onApr 28, 2020 at 7:30am PDT

And now, speaking to SpotBoyE, she has clarified no such thing has happened. She said, "I am in the mood of getting married but it's just that I haven't found the right person yet. Mere dil ki ghanti nahi baji ab tak." She added, "Ever since I posted that picture, people are sending me congratulatory messages. Many friends are even wishing me 'A Happy Married Life'. In fact, that very day, Kashmera Shah texted me on similar lines."

So what was the secret behind that picture? She spilled the beans and said, "I was shooting a music video on the song Chupke Se from Saathiya at home. Yes, I dressed like a newly-wed girl at her honeymoon. But surely, I didn't expect that it would lead to so much chaos."

And here is the video she is talking about, watch it if you haven't:

 
 
 
View this post on Instagram

Chupke se ❤️🤫🤫

A post shared by Jasleen Matharu ਜਸਲੀਨ ਮਠਾੜੂ (@jasleenmatharu) onApr 28, 2020 at 6:54am PDT

But it seems fans are really wanting her to turn a bride soon and let's see when that happens.

Catch up on all the latest entertainment news and gossip here. Also, download the new mid-day Android and iOS apps.

Mid-Day is now on Telegram. Click here to join our channel (@middayinfomedialtd) and stay updated with the latest news




etl

Millions affected, billions at stake: Scientists urge ‘wiser’ use of wetlands to tackle poverty and conserve ecosystems

2nd February 2014 – Colombo, Sri Lanka Agriculture and wetlands should be managed in unison in order to conserve vital ecosystems and support the livelihoods of millions of people, according to a new report published to coincide with World Wetlands Day, today. Download the full media release Download the report




etl

DailyMirror: To help an Earth under stress, let’s look to Sri Lanka’s wetlands

With Earth Day marked on April 22, we look to nature’s solutions to climate change and other challenges. Wetland preservation is vital for our environmental, food and societal futures.




etl

DailyMirror: To help an Earth under stress, let’s look to Sri Lanka’s wetlands

With Earth Day marked on April 22, we look to nature’s solutions to climate change and other challenges. Wetland preservation is vital for our environmental, food and societal futures.




etl

Planned Parenthood Sues Anti-Abortion Group for Filming It Secretly

The conservative Center for Medical Progress in 2015 released video it secretly filmed, purporting to show Planned Parenthood officials haggling over the sale price for aborted human fetuses.




etl

Chinese Collect User Data from random Android Phones secretly: Security Firm

Virginia-based security firm Kryptowire recently reported that there are few Android devices from a Chinese company that are infected with software that collects users’ personal information and conversation made through text messages and calls.




etl

AC Milan had secretly purchased a genius sign secret treaty

Although he captained the Champions League plus the play-off wholesale nike jerseys Champions League, but night AC Milan visit PSV,...




etl

ETL testing

What is ETL Testing?

A database is one of the most important assets an organization may own, and the data contained within that database is generally invaluable. It is common now for an...




etl

ETL Development (Microstrategy Developer)

Company: FIS Global Business Solutions India Private Limited
Experience: 1 to 6
location: India
Ref: 24827886
Summary: Job Description : Position Type : Full time Type Of Hire : Experienced (relevant combo of work and education) Education Desired : Bachelor of Computer Science Travel Percentage : 25% Are you curious, motivated, ....




etl

Sr.. ETL Developer (Talend, AWS cloud).

Company: Datamine Infotech
Experience: 4 to 8
location: Bengaluru / Bangalore
Ref: 24822481
Summary: Need to interface with business users, Source SMEs and identify all the functional data requirements. Prepare High Level Design and ETL design. Create optimized logical and physical data models aligned with all key DWH modelling....




etl

Tennis bad boy Nick Kyrgios uses a toilet break to secretly smash two racquets before ugly meltdown 

The Australian tennis star could face suspension after calling an umpire a 'f***ing tool' and appearing to spit in his direction at the Cincinnati Masters on Wednesday.




etl

Danielle Brooks sweetly cradles her baby bump in pink and red dress

She announced she is expecting her first child on a July 2 Instagram post in partnership with Clear Blue. And on Friday morning, Danielle Brooks was spotting cradling her baby bump in NYC.




etl

Bull charges out of a public toilet on the Shetland Islands

The bull had escaped from a nearby farm and briefly hid out in a gents toilets on the Shetland Islands in Scotland before deciding to make a break for it.




etl

Up Helly Aa 2019: Thousands dressed as Vikings descend on Shetland

The Up Helly Aa fire festival, which attracts visitors from around the globe, takes place in Lerwick on the last Tuesday of January each year.




etl

Rand Paul accuses his detractors of 'secretly' wishing for a terrorist attack

'People here in town think I’m making a huge mistake,' he said Sunday. 'Some of them, I think, secretly want there to be an attack on the United States so they can blame it on me.'




etl

Emma Stone sparks fan speculation she is secretly MARRIED

Emma Stone has sparked speculation she has secretly married fiancé Dave McCary.




etl

X-Factor semi-finalist Danny Tetley ADMITS he sexually exploited SEVEN teenage boys

Danny Tetley (pictured), from Bradford, West Yorkshire, pleaded guilty to getting his victims, many as young as 14, to send him explicit images of themselves.




etl

The Irishman review: Robert De Niro is fabulous and Joe Pesci quietly almost as good

The Irishman is three-and-a-half hours long and is out on Netflix towards the end of the month, raising the initially tempting prospect of watching it at home, maybe spread over two, even three nights.




etl

Laura Trott won a 50-lap race while secretly pregnant

Pregnant Olympic champion Laura Kenny, 25, revealed she rode 50 laps to win the 2016 British Cycling National Womens' Madison Championships as she appeared on GMB.




etl

Adau Mornyang was locked up after 'illegally' entering the US and has quietly returned home

Adau Mornyang, 25, was held in an Immigration and Customs Enforcement detention centre in New York for three months before her release on December 21.




etl

Victoria Beckham pokes fun at son Brooklyn's 'dodgy top' as he sweetly hugs sister Harper

The matriarch, 45, shared a sweet snap of Brooklyn, 20, hugging little sister Harper, eight, but couldn't help but mock the aspiring photographer's 'dodgy' fashion choice.




etl

Victoria Beckham is caught secretly filming vain husband David achieve the 'perfect selfie'

On Sunday, it was Victoria's turn to poke fun at David, 44, as he was caught secretly attempting to achieve the 'perfect selfie' in a funny Instagram clip.




etl

Hire car driver secretly tuned in to Seal dissing Boy George

A hire car driver has revealed he secretly 'tuned in' to singer-songwriter Seal sharing his 'distaste for Boy George.'




etl

Gwen Stefani and Blake Shelton allegedly secretly married

Rumors are swirling about the status of Gwen Stefani and Blake Shelton's relationship. According to anonymous sources, the pair secretly tied the knot in a country-themed Los Angeles wedding in 2017.




etl

Treasury secretary Steve Mnuchin quietly meets RUSSIA'S ambassador while the markets plunge

Treasury Secretary Steve Mnuchin met with Russia's ambassador to the U.S. amid the steep drop in financial markets Monday.




etl

Donald Trump surrogate is quietly courting Muslims to downplay controversy over ban  

Walid Phares, a top national security adviser for Donald Trump, is quietly talking to Muslims on behalf of the presumptive Republican nominee.




etl

Kate Hudson sweetly cradles her baby daughter Rani as she enjoys Italian family getaway

Kate Hudson, 40, was every inch the doting mum to her baby daughter Rani on Sunday, as she enjoyed a sun-soaked break to Italy with her family and boyfriend Danny Fujikawa, 32.




etl

Oscar Pistorius secretly FREED from jail to attend memorial service for his grandmother

The Olympian was given 'compassionate leave' to attend the memorial service because he is considered 'low risk' by bosses at South Africa's Kgosi Mampuru jail.




etl

Oscar Pistorius to be secretly freed from prison on compassionate leave

Former Paralympian Oscar, 31, is serving a 15-year-sentence for the murder of his model girlfriend Reeva Steenkamp, 29, who he blasted to death in his own bathroom in South Africa.




etl

Go wild with water: Create your own wetland pond habitat and let nature flourish

British gardening expert Nigel Colborn shared advice for maintaining a healthy, clear-water pond with thriving wildlife. He recommends placing a site in part-shade or sun.




etl

Kylie Jenner and daughter Stormi sweetly sing happy birthday to True Thompson

Kylie Jenner and daughter Stormi adorably serenaded Khloé Kardashian and Tristan Thompson's little girl, as she rang in her second year of life, with a virtual performance of Happy Birthday.




etl

Emma Stone sparks fan speculation she is secretly MARRIED

Emma Stone has sparked speculation she has secretly married fiancé Dave McCary.




etl

Gwen Stefani sweetly signs leg of super-fan... right near detailed tattoo of her FACE

Gwen Stefani has inspired some serious fandom. And while hitting a red carpet for her Las Vegas show she came upon an admirer who had her face tattooed in detail on one of his calves.




etl

Taika Waititi and wife Chelsea Winstanley 'separated quietly two years ago'

New Zealand Thor director Taika Waititi and his film producer wife Chelsea Winstanley appear to have separated. 




etl

RAY MASSEY: California dreaming with the new Volkswagen Beetle Cabriolet

I love the look of the new Volkswagen Beetle Cabriolet which makes its world debut at the Los Angeles Motor Show on November 28 and will go on sale in Britain in spring.




etl

Smitten with the bug: RAY MASSEY tests the Volkswagen Beetle Cabriolet

It’s no slouch. On Germany’s unrestricted motorways, it’ll cruise happily and legally top-down in the ‘slow’ lane at 100 mph.




etl

Dad secretly gave his house to my sister 20 years ago - what can I do?

My brother's wife went to the Land Registry and found the deeds were put in my sister's name in 1999. As far I know there was no will. Ian Johnston (pictured), solicitor at JMW Solicitors, replies.




etl

Skye Wheatley fuels rumours she is secretly married

Instagram model Skye Wheatley, 25, has fuelled engagement rumours again this week by using the term 'husband' while referring to her beau Lachlan Waugh.




etl

Pregnant Skye Wheatley hints she's secretly married boyfriend Lachlan

The cryptic clue comes after she first started referring to Lachlan as her 'husband' in February.




etl

Alan Rickman has 'secretly wed Rima Horton 50 years after they first met'

The couple, who have been living together since 1977, married in a secret ceremony in New York recently, Alan, 69, has revealed to German newspaper Bild.




etl

Beyonce sweetly comforts Vanessa Bryant by holding onto arm at Kobe and Gianna's memorial service

The 38-year-old megastar singer's compassionate actions were captured momentarily during the telecast of the Celebration of Life for Kobe and Gianna Bryant on Monday.




etl

TONY HETHERINGTON: Your debtor's 800 miles away in Shetland

I have received a letter from Lowell Portfolio, claiming an alleged debt of £599 which it says I owe to PayPal. But I rang PayPal and was told that in fact no money is owed on my account.




etl

Caretaker, 42, is caught secretly filming women in primary school toilets

Matthew Stupples, 42, lived on-site at Wimbledon Park Primary School in south west London where he had worked as a caretaker and handyman for nearly 15 years.




etl

David Beckham secretly films wife Victoria achieving the 'perfect' selfie as he watches The Crown

The former footballer, 44, shared the great lengths his wife Victoria, 45, goes to achieve the perfect selfie as he secretly filmed her attempts in a cheeky Instagram clip on Saturday.




etl

Flush your toilet with rain - and save bucketloads!

Making home improvements to save money is still possible with a little bit of imagination - and online ordering. Toby Walne looks at all your options.




etl

TONY HETHERINGTON: Your debtor's 800 miles away in Shetland

I have received a letter from Lowell Portfolio, claiming an alleged debt of £599 which it says I owe to PayPal. But I rang PayPal and was told that in fact no money is owed on my account.




etl

The Blue Lady anchors, quietly


Yet another ship with toxic waste has recently beached at Alang, Gujarat. The Blue Lady's owner admits that the ship contains asbestos. But the ship carries neither documents required as per international law, nor a complete inventory of its hazardous wastes, says Gopal Krishna.




etl

Maimed by the state, quietly


Amidst a culture of silence and media inattention, torture is easy to find in the security hot zones of India. A new film bares the ugly truth. Freny Manecksha reports.




etl

Maharashtra secretly amends RTI Rules


The State has quietly pushed through a set of changes to the way it treats RTI applications. Activists discovered it quite accidentally, and are shocked. Krishnaraj Rao writes.