How to use INCREMENTAL REFRESH for Datasets (PRO) and Dataflows (PREMIUM) in Power BI

great topic cover,today,thanks ,making in dept topic

Hi,
My "Apply All" button is greyed out.. I can't click it... What could be the issue? Thanks...

It would be good, if you can show any data, that was done by incremental refresh.

Hi, How to know if my incremental refresh is working in DataFlow?

With Power BI Desktop I can connect in SSMS with EndPoint XML, but how to do this with DataFlows?

Can you help me please ?

Very timely video
I was able to follow all instructions
Thanks for making it seem so easy for me to understand
I tried refreshing my datasets though but still it refreshes my entire dataset as oppose to refrshing the new loaded data only

My source is excel from sharepoint

I love this guy! Unlike that Yoooo one from guy in cube! He good too but this guy is to the point from the first second of the video!
Hugs and kisses!

Mankasitraka e! quite helpful video. thanks for sharing!!!

Very good video. The bit I am confused about is how it uses the 'order date' field to limit which rows it refreshes if that order date field only exists after the data has been transformed. For example my dates come in as a number (i.e. 9945684) which I have to convert into date format, so how does it decide if a row needs to be refreshed before it refreshes it? Does it look at the data it has gathered in a previous refresh and use that to exclude which rows are refreshed? Or does it apply the transformations to a single date column for all rows and use this to limit the refresh? Or am I missing something completely?

Many thanks for the very detailed explanation!
I have another problem with the Inc. Refresh: storing the last 7 business days & only refreshing the last 1. When the data gets refreshed, it duplicates.
Any one has encountered this? Any solution?

Hi Fernan, very useful video. I need your advise as facing one issue as I set up incremental refresh but after every scheduled or on demand refresh it is not overridding the parameters. Hence, not able to load recent data instead getting same(OLD) data from RangeStart till RangeEnd. Any solution?

do i have to create dataflow to configure or enablle incremental refresh on power bi cloud service

Great video.

Thanks for showing this. I got M query fold error after apply date filter. any Suggestions on fixing it?

Hi, Thank you for the video. In my case i am using more than one table in the dataset. in that case do we need to follow the same process for all the tables to setup incremental refresh in powerbi desktop? Also one of my report it uses past 20 years data to show the report values and monthly refresh needed. when i filter the data as you suggest the data got filterered and i am not seeing the needed output.. if it possible to use incremental refresh when we show data in the visual using past 20 years data.

You're using OrderDate both for the RangeStart / RangeEnd filtering on the table, and for the Detect Data Changes field. The documentation explicitly says you shouldn't do this, and that Detect Data Changes should be set on a different field: "This should not be the same column used to partition the data with the RangeStart and RangeEnd parameters." The docs don't clarify why it's a problem (e.g. what will go wrong), but I'm wondering if your example in the video might encounter issues because it's not following this guideline. Thoughts?

Hi, in my query editor i have one main table "Raw Data" linked to the database, i have created one referenced table by using Main table which is named as "RawData2". now for incremental refresh, do i need to follow the process for both the tables or just main table.
i am asking this because i believe reference table also taking data directly from database not from the main data.

Great Video Fernan

Hi , very nice video and informative but I have one question. In case of a dataset , and in table that it’s m query has many steps in which step I had to apply the custom filter? At top after the step is calling the source ? or at the end? I tried the second option and the performance of refresh is 5x worse than before

That's a great video. Thank you so much for explaining this in detail

Thanks for the video! How do we handle when in my SQL database the table has changed (new column, deleted column) and want to exposed those changes? Does this mean we have create a new power bi file and do a full refresh? Is there a way to avoid a full refresh?

Thank you for your video. I am new to use this. sorry if it is a silly doubt. I have a doubt, I have csv files in sharepoint folder. where every week a new csv file is uploaded to it. I created a data flow to save one of these csv file data in dataverse using dataflow in dataverse. Now in order to update following weeks csv files, I want to use incremental refresh. to do that we create RangeStart and RangeEnd in dataflow of dataverse. Now how to go to incremental refresh in dataflow in dataverse. Is incremental refreh available only in power BI editor? show I definatly create workspace and datflow in power BI?

For me its showing Premium Only, Even though I have premium.

Hi Fernan for this awesome Tutorial.
I need advise from you.
I have a dataset in BigQuery that changes every minute (new rows gets added every minute). The size of the dataset is around 13 GB. I need to find way to refresh my data every minute, but not the whole data set. As far as I understood, incremental refresh setting does not refresh the data every minute. Is there a way I can solve my issue?

I have a fact table with start_year (2020) and end_year(2023) parameters in it to limit the number of years worth of data. On top of this I have incremental refresh of last 13 months. I publish this file to web service and change the start_year parameter say from 2020 to 2019. Why does the parameter changes in web service doesn't take into effect? If I change from 2020 to 2021, it does make necessary changes but for the first time only after publishing the file. Any thought on this? My report should only hold only last 5 years of data with last 13 months of incremental refresh.

How does this work when your source is on VM (virtual machine server) or Dataflows?

thanks for the content fernan, i have one question which is very confusing to me. some vids on the web says you need directquery / queryfolding to use incremental refresh logically otherwise with import mode your incremental refresh is meaningless. however on incremental refresh page powerbi says "you need import mode to apply incremental refresh" i got confused. could you help me?

I have a staging workspace where I set up a dataflow that imports data from an on-prem SQL Server database. On one table I set up an incremental refresh and it takes 30 seconds to update.

In a separate workspace dataflow, I linked to that same incrementally refreshed table. However, it takes 3 minutes to refresh.

The linked table has a Referenced table associated with it. Do I need set up an incremental refresh on the Referenced table, too?

this only works for 'date/time' selected, 'date' doesn't work. so you really have to follow the steps to get incremental refresh running.

i load data from excel my new excel file contain only additional data that i want to append with existing on ... is there any way to do this

Can I use RangeStart and RangeEnd data type as Date as I have created date as date only? Also my dataset is in import mode and source is mysql db is it possible?

Thanks, Fenan for this wonderful tutorial, quick question, will the Refresh button be clicked on Power BI service or Power BI desktop for the incremental refresh to work?

In order to understand incremental refresh, we need to be able to understand answers to these questions.
1. What is incremental refresh?
2. Why do we need it or what is the benefit of using this?
3. What are the license requirements to use it?
4. What are the draw backs of using it? What are its limitations and such?
5. Which fields are we considering for incremental refresh if there are more than one date field in the dataset?
6. What if dataset has date instead of date time? Will it create an issue?
And, there might be more unanswered questions to it which I have not mentioned above on my lists.
Thanks

can we do it in the connect level by puttin a where clause in the sql

Amazing

I have followed all the steps but when I check the next day it didn't refresh. and when we give incremental refresh to desktop will it not allow to refresh in services?

my "parameter" is dimmed in filter rows. is it because im not premium license?

thank you. this is a very helpful video!

Thanks Fernan!

Hi Dear,
I have power query table called "tbl_1".
I need to have another table called "tbl_2" sourcing data at "tbl_1" step no. 4.
There are other steps also after step#4. I don't want to duplicate table because it was decrease its efficiency. How can it be achieved without duplicating table?

But how could we automate the RangeStart and RangeEnd parameters ?

If I am not using any database and my power query extracted data directly from share point or an excel file, will I be able to do incremental refresh as well? bcoz my query now tooks almost 2 hours to complete refresh

This helped me a lot. Thank you!!!

How can I highlight the data that have been changed after the update?