Our customers’ tests have become more complex, longer, and consume more resources than before. This increases the need to optimize the regression while not compromising on coverage. Some advanced...

[[ Click on the title to access the full blog on the Cadence Community site. ]]

In the former blog about Python and Specman: Specman: Python Is here!, we described the technical information around Specman-Python integration. Since Python provides so many easy to use existing libraries in various fields, it is very tempting to leverage these cool Python apps.

Coverage has always been the center of the verification methodology, however in the last few years it gets even more focus as people develop advanced utilities, usually using Machine Learning aids. Anyhow, any attempt to leverage your coverage usually starts with some analysis of the behavior and trends of some typical tests. Visualizing the data makes it easier to understand, analyze, and communicate. Fortunately, Python has many Visualization libraries.

In this blog, we show an example of how you can use the plotting Python library (matplotlib) to easily display coverage information during a run. In this blog, we use the Specman Coverage API to extract coverage data, and a Python module to display coverage grades interactively during a single run and the way to connect both.

Before we look at the example, if you have read the former blog about Specman and Python and were concerned about the fact that python3 is not supported, we are glad to update that in Specman 19.09, Python3 is now supported (in addition to Python2).

The Testcase
Let’s say I have a stable verification environment and I want to make it more efficient. For example: I want to check whether I can make the tests shorter while hardly harming the coverage. I am not sure exactly how to attack this task, so a good place to start is to visually analyze the behavior of the coverage on some typical test I chose. The first thing we need to do is to extract the coverage information of the interesting entities. This can be done using the old Coverage API. 

Coverage API
Coverage API is a simple interface to extract coverage information at a certain point. It is implemented through a predefined struct type named user_cover_struct. To use it, you need to do the following:

1. Define a child of user_cover_structusing like inheritance (my_cover_struct below).
2. Extend its relevant methods (in our example we extend only the end_group() method) and access the relevant members (you can read about the other available methods and members in cdnshelp).
3. Create an instance of the user_cover_structchild and call the predefined scan_cover() method whenever you want to query the data (even in every cycle). Calling this method will result in calling the methods you extended in step 2.  

 The code example below demonstrates these three steps. We chose to extend the end_group() method and we keep the group grade in some local variable. Note that we divide it by 100,000,000 to get a number between 0 to 1 since the grade in this API is an integer from 0 to 100,000,000. 

Pass the Data to a Python Module
After we have extracted the group grade, we need to pass the grade along with the cycle and the coverage group name (assuming there are a few) to a Python module. We will take a look at the Python module itself later. For now, we will first take a look at how to pass the information from the e code to Python. Note that in addition to passing the grade at certain points (addVal method), we need an initialization method (init_plot) with the number of cycles, so that the X axis can be drawn at the beginning, and end_plot() to mark interesting points on the plot at the end. But to begin with, let’s have empty methods on the Python side and make sure we can just call them from the e code.

 # plot_i.py
def init_plot(numCycles):
    print (numCycles)
def addVal(groupName,cycle,grade):
    print (groupName,cycle,grade)
def end_plot():
    print ("end_plot") 

And add the calls from e code:

 Before running this, note that you need to ensure that Specman finds the Python include and lib directories, and Python finds our Python module. To do this, you need to define a few environment variables: SPECMAN_PYTHON_INCLUDE_DIR, SPECMAN_PYTHON_LIB_DIR, and PYTHONPATH. 

 The Python Module to Draw the Plot
After we extracted the coverage information and ensured that we can pass it to a Python module, we need to display this data in the Python module. There are many code examples out there for drawing a graph with Python, especially with matplotlib. You can either accumulate the data and draw a graph at the end of the run or draw a graph interactively during the run itself- which is very useful especially for long runs.

Below is a code that draws the coverage grade of multiple groups interactively during the run and at the end of the run it prints circles around the maximum point and adds some text to it. I am new to Python so there might be better or simpler ways to do so, but it does the work. The cool thing is that there are so many examples to rely on that you can produce this kind of code very fast.

 In the example we used, we had two interesting entities: packet and state_machine, thus we had two equivalent coverage groups. When running our example connecting to the Python module, we get the following graph which is displayed interactively during the run.

When analyzing this specific example, we can see two things. First, packet gets to a high coverage quite fast and significant part of the run does not contribute to its coverage. On the other hand, something interesting happens relating to state_machine around cycle 700 which suddenly boosts its coverage. The next step would be to try to dump graphic information relating to other entities and see if something noticeable happens around cycle 700.

To run a complete example, you can download the files from: https://github.com/okirsh/Specman-Python/

Do you feel like analyzing the coverage behavior in your environment? We will be happy to hear about your outcomes and other usages of the Python interface.

Orit Kirshenberg
Specman team

Our customers’ tests have become more complex, longer, and consume more resources than before. This increases the need to optimize the regression while not compromising on coverage.

Some advanced customers of Specman use Machine Learning based solutions to optimize the regressions while some use simpler solutions. Based on a request of an advanced customer, we added a new Coverage API in Specman 19.09 called Coverage Callback. In 20.03, we have further enhanced this API by adding more options. Now there are two Coverage APIs that provide coverage information during the run (the old scan_cover API and this new Callback API). This blog presents these two APIs and compares between them while focusing on the newer one.

Before we get into the specifics of each API, let’s discuss what is common between these APIs and why we need them. Typically, people observe the coverage model after the test ends, and get to know the overall contribution of the test to the coverage. With these two APIs, you can observe the coverage model during the test, and hence, get more insight into the test progress.

Are you wondering about what you can do with this information? Let’s look at some examples.

1. Recognize cases when the test continues to run long after it already reached its coverage goal.
2. View the performance of the coverage curve. If a test is “stuck” at the same grade for a long time, this might indicate that the test is not very good and is just a waste of resource.

These analyses can be performed in the test itself, and then a test can decide to either stop the run, or change something in it configuration, or – post run. You can also present them visually for some analysis, as shown in the blog: Analyze Your Coverage with Python.

scan_cover API (or “Scanning the Coverage Model”)

With this API you can get the current status for any cover group or item you are interested in at any point in time during the test (by calling scan_cover()). It is very simple to use; however it has performance penalty. For getting the coverage grade of any cover group during the test, you should
1. Trigger the scan_cover at any time when you want the coverage model to be scanned.
2. Implement the scan_cover related methods, such as start_item() and end_bucket(). In these methods, you can query the current grade of group/item/bucket.
The blog mentioned earlier: Analyze Your Coverage with Python describes the details and provides an example.

Callback API

The Callback API enables you to get a callback for a desired cover group(s), whenever it is sampled. This API also provides various query methods for getting coverage related information such as what the current sampled value is. So, in essence, it is similar to scan_cover API, but as the phrase says: “same same but different”:

1. Callback API has almost no performance penalty while scan_cover API does.
2. Callback API contains a richer set of query methods that provide a lot of information about the current sampled value (vs just the grade with scan_cover).
3. Using scan_cover API, you decide when you want to query the coverage information (you call scan_cover), while with the Callback API you query the coverage information when the coverage is sampled (from do_callback). So, scan_cover gives you more flexibility, but you do need to find the right timing for this call.

There is no absolute advantage of either of these APIs, this only depends on what you want to do.  

Callback API details

The Callback API is based on a predefined struct called: cover_sampling_callback. In order to use this API, you need to:

1. Define a struct inheriting cover_sampling_callback (cover_cb_save_data below)
   1. Extend the predefined do_callback() method. This method is a hook being called whenever any of the cover groups that are registered to the cover_sampling_callback instance is being sampled.
   2. From do_callback() you can access coverage data by using queries such as: is_currently_per_type(), get_current_group_grade() and get_current_cover_group() (as in the example below) and many more such as: get_relevant_group_layers() and get_simple_cross_sampled_bucket_name().
2. Register the desired cover group(s) to this struct instance using the register() method.

Take a look at the following code:

// Define a coverage callback.
// Its behavior – print to screen the current grade.
struct cover_cb_save_data like cover_sampling_callback {
    do_callback() is only {
       // In this example, we care only about the per_type grade, and not per_instance
       if is_currently_per_type() {           
            var cur_grade : real = get_current_group_grade();
            sys.save_data (get_current_cover_group().get_name(), cur_grade);
        };//if
    };//do_callback()
};// cover_cb_send_data


extend sys {
    !cb : cover_cb_save_data;

   // Instantiate the coverage callback, and register to it two of my coverage groups
    run() is also {
       cb  = new  with {
        var gr1:=rf_manager.get_struct_by_name("packet").get_cover_group("packet_cover");
        .register(gr1);
        var gr2:=rf_manager.get_struct_by_name("sys").get_cover_group("mem_cover");
       .register(gr2);
       };//new  
    };//run()

  save_data(group_name : string, group_grade : real) is {
        //here you either print the values to the screen, update a graph you show or save to a db 
  };// save_data
};//sys

In the blog Analyze Your Coverage with Python mentioned above, we show an example of how you can use the scan_cover API to extract coverage information during the run, and then use the Specman-Python API to display the coverage interactively during the run (using plotting Python library - matplotlib). If you find this usage interesting and you want to use the same example, by the Callback API instead of the scan_cover API, you can download the full example from GIT from here: https://github.com/efratcdn/cover_callback.

Specman Team

Hello!

I have input signal like this  ->  input  wire [ADM_NUM-1:0][1:0] m_axi_ddr_rresp.

When i want to analyze coverage from IMC  this signal not covered!

Can i collect coverage for this signal?

I have a problem connected with my AXI4 coverage.

I enable coverage collection in AXI4 

      set_config_int("axi4_active_slave_agent_0.monitor.coverModel", "burst_started_enable", 1);
      set_config_int("axi4_active_slave_agent_0.monitor.coverModel", "coverageEnable", 1);

but i don't have a result.

I think the problem in Callback, but i try to connect all callback and i don't have positive result.

Can you help me?

Hi  all,

          I am getting this warning in while generating the coverage report, can you help me to clear this warning?

ncsim: *W,COVOPM: Coverage configuration file command "set_covergroup -optimize_model" can be specified to improve the performance and scalability of coverage model containing SV covergroups. It may be noted that subsequent merging of a coverage database saved with this command and a coverage database saved without this command is not allowed.

I'm in the process of weeding a regression test list. I have a coverage database from the full regression list and would like to diff it with the coverage database from the new reduced regression test list. If possible I would than like to trace back any buckets covered with the full list, but not with the partial list, into the original tests that covered them.

Is that possible using IMC? if not, is it possible to do from Specman itself?

(Note that we're not using vManager)

Thanks,

Avidan

I want to write a transition coverage on an enumeration. One of the parts of that transition is a queue of the enum. I construct this queue in my constructor. Considering the example below, how would one go about it.

In my coverage bin I can create a range like this A => [queue1Enum[0]:queue1Enum[$]] => [queue2Enum[0]:queue2Enum[$]]. But I only get first and last element then.

typedef enum { red, d_green, d_blue, e_yellow, e_white, e_black } Colors;
 Colors dColors[$];
 Colors eColors[$];
 Lcolors = Colors.first();
 do begin
  if (Lcolors[0].name=='d') begin
   dColors.push_back(Lcolors);
  end
  if (Lcolors[0].name=='e') begin
   eColors.push_back(Lcolors);
  end
 end while(Lcolors != Lcolors.first())

 covergroup cgTest with function sample(Colors c);
   cpTran : coverpoint c{
      bins t[] = (red => dColors =>eColors);   
   }
 endgroup

વિશાખાપટ્ટનમ ખાતે ગુરુવારે થયેલી ગેસ લીકની દુર્ઘટનાએ ઇતિહાસની સૌથી મોટી ભોપાલ ગેસ દુર્ઘટનાની સ્મૃતિ તાજા કરી દીધી. દુનિયાએ જોયેલી કેટલીક આવી જ ઔદ્યોગિક દુર્ઘટનાઓ પર એક નજર કરીએ...

This Metasploit module exploits an arbitrary file upload vulnerability in Numara / BMC Track-It! v8 to v11.X. The application exposes the FileStorageService .NET remoting service on port 9010 (9004 for version 8) which accepts unauthenticated uploads. This can be abused by a malicious user to upload a ASP or ASPX file to the web root leading to arbitrary code execution as NETWORK SERVICE or SYSTEM. This Metasploit module has been tested successfully on versions 11.3.0.355, 10.0.51.135, 10.0.50.107, 10.0.0.143, 9.0.30.248 and 8.0.2.51.

XMB - eXtreme Message Board version 1.9.11.13 suffers from weak crypto and insecure password storage vulnerabilities.

Whitepaper called Android Security Research: Crypto Wallet Local Storage Attack.

US web infrastructure and cyber security company Cloudflare wants to improve people’s internet experiences through affordable, reliable and accessible interconnection points, especially in less privileged parts of the world. 

Siemens Gamesa Renewable Energy (SGRE) said that is has begun operation of its electric thermal energy storage system (ETES), a milestone in the development of energy storage solutions, according to the company.

Natron Energy said that the California Energy Commission (CEC) awarded it a $3 million grant for “Advanced Energy Storage for Electric Vehicle Charging Support.” Natron will use the money to manufacture and install a high powered, long cycle life energy storage system at an EV Fast Charging station.

Iowa economic development officials are tentatively endorsing a tax credit for battery storage to complement the state’s wind and solar generation.

U.S. Sens. Ron Wyden (D-Ore.), Jeff Merkley (D-Ore.), Angus King (I-Maine), Brian Schatz (D-Hawaii), and Jack Reed (D-R.I.), have reintroduced The Marine Energy Research and Development Act, intended to increase production of low-carbon, renewable energy from the natural power in ocean waves, tides and currents.

Go Electric Inc. is a specialist in energy resiliency solutions for microgrids and commercial & industrial customers integrating renewables and generators

Eos Energy Storage is manufacturer and supplier of the zinc-based Aurora 2.0 battery system for Duke Energy's McAlpine substation and as a behind-the-meter solution at the University of California, San Diego. 

8minute Solar Energy, NV Energy and the Moapa Band of Paiutes announced that NV Energy selected 8minute to develop the largest solar plus storage project ever built in Nevada and one of the largest in the world.

Yesterday, ENGIE Storage announced that San Diego International Airport (SAN) installed a 2 MW/4 MWh GridSynergy energy storage system. Paired with the airport’s existing 5.5 MW of solar capacity, the new energy storage system will reduce energy charges during peak demand, which according to ENGIE equate to approximately 40 percent of the airport’s monthly electricity costs. The system is expected to begin operation in early 2020.

On July 1, Public Service of New Mexico filed a plan with regulators in the state for how it plans to get to a 100 percent emission-free power by 2040. The utility reviewed four scenarios, all of which involved the early retirement of the San Juan Coal Plant, to arrive at its recommended path forward.

Commercial property owners with existing energy storage systems, or owners considering implementing an energy storage system, may be able to benefit from a recent order by the Massachusetts Department of Public Utilities (DPU) allowing utility companies to pay customers who agree to rely upon their energy storage systems and dispatch the energy during peak events.

In April 2019, in the heart of coal country, Indiana regulators rejected a proposal by its electric and gas utility, Vectren, to replace baseload coal plants with a new $900 million, 850 megawatt (MW) natural gas-fired power plant. Regulators were concerned that with the dramatic decline in the cost of renewable energy, maturation of energy storage and rapidly changing customer demand, such a major gas plant investment could become a stranded, uneconomic asset in the future. Regulators are now pushing Vectren to consider more decentralized, lower-carbon resources such as wind, solar and storage that would offer greater resource diversity, flexibility and cost effectiveness.

This week artificial intelligence (AI)-driven energy storage services provider Stem said that it had formed a partnership with New York-based private equity company Syncarpha Capital to build 28.2 megawatt-hours (MWh) of large-scale storage projects co-sited with solar in Massachusetts.

Voith has received an order for the Ritom pumped storage power plant in Switzerland, which began operating in 1920 and will be replaced with a new facility.

Gas peaker plants may be among the first casualties of a new Minnesota law requiring utilities to include energy storage as part of their long-range plans.

Finnish energy technology group Wärtsilä has signed an EPC contract for a 100 MW/100 MWh total capacity energy storage project in southeast Asia.

Last week, California’s Glendale Water & Power (GWP) received approval from the Glendale City Council to move forward with a plan to repower the aging Grayson Power Plant with a combination of renewable energy resources, energy storage and a limited amount of thermal generation.

According to the latest forecast by BloombergNEF (BNEF), energy storage installations (not including pumped hydropower) around the world will multiply exponentially, from 9GW/17GWh deployed as of 2018 to 1,095GW/2,850GWh by 2040.

ENGIE Storage has announced it will supply and operate a 19 MW/38 MWh portfolio of six energy storage sites that will contribute to the Solar Massachusetts Renewable Target Program and participate in ISO-New England wholesale markets.

The Skeleton Creek project will be located in three Oklahoma counties

Dominion executives said in an interview this week that battery storage has the potential to improve the resiliency of the electrical grid

The four proposed central Virginia lithium-ion projects will cost about $33 million to build and offer 5-years’ worth of test data, according to Dominion.

A report recently released by the U.S. Department of Energy defines and evaluates cost and performance parameters of six battery energy storage technologies (BESS) and four non-BESS storage technologies.

South Australian energy storage company 1414 Degrees and Stone & Wood Brewing Company have agreed to undertake a feasibility study for the integration of 1414 Degrees’ electrically charged Thermal Energy Storage System (TESS-IND).

This week, San José’s Community Choice Aggregator (CCA) which is called San José Clean Energy (SJCE) and EDP Renewables SA (EDPR), through its fully owned subsidiary EDP Renewables North America LLC (EDPR NA), signed a 20-year power purchase agreement (PPA) for 100 MW of new solar energy capacity and 10 MW of battery storage at the Sonrisa Solar Park in Fresno County, California. The project is anticipated to be operational in 2022.

Energy Vault, the creator of a new way to perform utility-scale energy storage, announced that SoftBank Vision Fund (Vision Fund) invested $110 million in its Series B funding round. Energy Vault will use the funds to accelerate global deployment of its technology, which enables renewables to deliver baseload power for less than the cost of fossil fuels 24 hours a day, said the company. As part of the investment, Andreas Hansson, Partner for SoftBank Investment Advisers, will join the Energy Vault board of directors.

Hydro Tasmania says it is accelerating detailed investigation of three key opportunities for pumped storage hydro development in the state as part of the Battery of the Nation initiative. Work has begun on a full feasibility assessment of pumped hydro development opportunities at Lake Cethana and Lake Rowallan in the northwest and near Tribute Power Station on the West Coast.

California’s energy storage incentive program has been a great success, with more than 11,000 battery storage systems installed to-date. The problem is, it’s not reaching the state’s most vulnerable communities. A new proposal from the California Public Utilities Commission (CPUC) aims to fix some of the barriers preventing disadvantaged communities from participating in the program, and it allocates $100 million to a new program designed to offset the cost of battery storage systems for populations threatened by wildfires and related utility power shutoffs.

Under a 20-year agreement, AES will provide APS with the storage capabilities powered by Fluence’s Advancion platform.

Unusually still weather in the upper Midwest and Great Plains in late 2018 has already taken a bite out of earnings at NextEra Energy Inc. and Avangrid Inc., which both operate large wind farms. Other wind generators have yet to report fourth-quarter results, including Pattern Energy Group Inc., TerraForm Power Inc. and Clearway Energy Inc.

The Lithuanian Business Support Agency (LSBA) has granted €235,000 (US$267,500) to support development of an experimental floating solar photovoltaic power plant at the existing 900-MW Kruonis pumped-storage hydroelectric plant in Lithuania.

An energy storage system has been installed at the largest single-site solar park in the world.

The unit of NextEra Energy announced plans to build a 409-MW energy storage facility in Manatee County. FPL says the Manatee Energy Storage Center will be the world’s largest solar-power battery system by four-fold.

After announcing nearly a gigawatt of new clean-energy projects in February, APS is now seeking proposals to build two of them in different parts of the state. The first request for proposals (RFP) seeks partners to add batteries to existing APS solar plants in rural Arizona, storing their power for use after the panels stop producing each day. A second partner is being sought to develop a large (100-MW) solar plant paired with an equal amount of storage, to bring more clean energy to customers after dark. Both of these projects will provide APS customers with more solar after sunset, serving their evening energy needs with an even cleaner resource mix.

ComEd sees a significant role for energy storage on Illinois’ electric grid as the state works toward realizing its ambitious renewable goals.

Yesterday, energy storage company Saft said it delivered and installed a turnkey Energy Storage System to Bermuda Electric Light Company (BELCO). The system provides up to 10 MW power for spinning reserves and frequency response to maintain grid stability and has a storage capacity of 5.5MWh, said the company. The official ribbon-cutting ceremony was held on May 16.

Iowa economic development officials are tentatively endorsing a tax credit for battery storage to complement the state’s wind and solar generation.