Neural network learning approach for color image restoration has been discussed in this paper and one of the possible solutions for restoring images has been presented. Here neural network weights are considered as regularization parameter values instead of explicitly specifying them. The weights are modified during the training through the supply of training set data. The desired response of the network is in the form of estimated value of the current pixel. This estimated value is used to modify the network weights such that the restored value produced by the network for a pixel is as close as to this desired response. One of the advantages of the proposed approach is that, once the neural network is trained, images can be restored without having prior information about the model of noise/blurring with which the image is corrupted.

In order to further improve the teaching quality of multimedia teaching in school daily teaching, a classroom facial expression emotion recognition model is proposed based on convolutional neural network. VGGNet and CliqueNet are used as the basic expression emotion recognition methods, and the two recognition models are fused while the attention module CBAM is added. Simulation results show that the designed classroom face expression emotion recognition model based on V-CNet has high recognition accuracy, and the recognition accuracy on the test set reaches 93.11%, which can be applied to actual teaching scenarios and improve the quality of classroom teaching.

Food quality is defined as a collection of properties that differentiate each unit and influences acceptability degree of food by users or consumers. Owing to the nature of food, food quality prediction is highly significant after specific periods of storage or before use by consumers. However, the accuracy is the major problem in the existing methods. Hence, this paper presents a BEFA_DRNN approach for accurate food quality prediction using dairy products. Firstly, input data is fed to data normalisation phase, which is performed by min-max normalisation. Thereafter, normalised data is given to feature fusion phase that is conducted employing DNN with Canberra distance. Then, fused data is subjected to data augmentation stage, which is carried out utilising oversampling technique. Finally, food quality prediction is done wherein milk is graded employing DRNN. The training of DRNN is executed by proposed BEFA that is a combination of BES and FA. Additionally, BEFA_DRNN obtained maximum accuracy, TPR and TNR values of 93.6%, 92.5% and 90.7%.

The supply chains of the Canadian forest products industry are largely dependent on accurate demand forecasts. The USA is the major export market for the Canadian forest products industry, although some Canadian provinces are also exporting forest products to other global markets. However, it is very difficult for each province to develop accurate demand forecasts, given the number of factors determining the demand of the forest products in the global markets. We develop multi-layer feed-forward artificial neural network (ANN) models for demand forecasting of the Canadian forest products industry. We find that the ANN models have lower prediction errors and higher threshold statistics as compared to that of the traditional models for predicting the demand of the Canadian forest products. Accurate future demand forecasts will not only help in improving the short-term profitability of the Canadian forest products industry, but also their long-term competitiveness in the global markets.

To better explore the application of unsupervised learning neural networks in psychological interventions for college students, this study investigates the relationships among latent psychological variables from the perspective of neural networks. Firstly, college students' psychological crisis and intervention systems are analysed, identifying several shortcomings in traditional psychological interventions, such as a lack of knowledge dissemination and imperfect management systems. Secondly, employing the Human-Computer Interaction (HCI) approach, a structural equation model is constructed for unsupervised learning neural networks. Finally, this study further confirms the effectiveness of unsupervised learning neural networks in psychological interventions for college students. The results indicate that in psychological intervention for college students. Additionally, the weightings of the indicators at the criterion level are calculated to be 0.35, 0.27, 0.19, 0.11 and 0.1. Based on the results of HCI, an emergency response system for college students' psychological crises is established, and several intervention measures are proposed.

In order to shorten the integration and sharing time of English teaching resources, a MOOC English online and offline mixed teaching resource integration model based on convolutional neural networks is proposed. The intelligent integration model of MOOC English online and offline hybrid teaching resources based on convolutional neural network is constructed. The intelligent integration unit of teaching resources uses the Arduino device recognition program based on convolutional neural network to complete the classification of hybrid teaching resources. Based on the classification results, an English online and offline mixed teaching resource library for Arduino device MOOC is constructed, to achieve intelligent integration of teaching resources. The experimental results show that when the regularisation coefficient is 0.00002, the convolutional neural network model has the best training effect and the fastest convergence speed. And the resource integration time of the method in this article should not exceed 2 s at most.

Aim/Purpose: The primary purpose of this study is to provide a cost-effective and artificial intelligence enabled security solution for IoT enabled healthcare ecosystem. It helps to implement, improve, and add new attributes to healthcare services. The paper aims to develop a method based on an artificial neural network technique to predict suspicious devices based on bandwidth usage. Background: COVID has made it mandatory to make medical services available online to every remote place. However, services in the healthcare ecosystem require fast, uninterrupted facilities while securing the data flowing through them. The solution in this paper addresses both the security and uninterrupted services issue. This paper proposes a neural network based solution to detect and disable suspicious devices without interrupting critical and life-saving services. Methodology: This paper is an advancement on our previous research, where we performed manual knowledge-based intrusion detection. In this research, all the experiments were executed in the healthcare domain. The mobility pattern of the devices was divided into six parts, and each one is assigned a dedicated slice. The security module regularly monitored all the clients connected to slices, and machine learning was used to detect and disable the problematic or suspicious devices. We have used MATLAB’s neural network to train the dataset and automatically detect and disable suspicious devices. The different network architectures and different training algorithms (Levenberg–Marquardt and Bayesian Framework) in MATLAB software have attempted to achieve more precise values with different properties. Five iterations of training were executed and compared to get the best result of R=99971. We configured the application to handle the four most applicable use cases. We also performed an experimental application simulation for the assessment and validation of predictions. Contribution: This paper provides a security solution for the IoT enabled healthcare system. The architectures discussed suggest an end-to-end solution on the sliced network. Efficient use of artificial neural networks detects and block suspicious devices. Moreover, the solution can be modified, configured and deployed in many other ecosystems like home automation. Findings: This simulation is a subset of the more extensive simulation previously performed on the sliced network to enhance its security. This paper trained the data using a neural network to make the application intelligent and robust. This enhancement helps detect suspicious devices and isolate them before any harm is caused on the network. The solution works both for an intrusion detection and prevention system by detecting and blocking them from using network resources. The result concludes that using multiple hidden layers and a non-linear transfer function, logsig improved the learning and results. Recommendations for Practitioners: Everything from offices, schools, colleges, and e-consultation is currently happening remotely. It has caused extensive pressure on the network where the data flowing through it has increased multifold. Therefore, it becomes our joint responsibility to provide a cost-effective and sustainable security solution for IoT enabled healthcare services. Practitioners can efficiently use this affordable solution compared to the expensive security options available in the commercial market and deploy it over a sliced network. The solution can be implemented by NGOs and federal governments to provide secure and affordable healthcare monitoring services to patients in remote locations. Recommendation for Researchers: Research can take this solution to the next level by integrating artificial intelligence into all the modules. They can augment this solution by making it compatible with the federal government’s data privacy laws. Authentication and encryption modules can be integrated to enhance it further. Impact on Society: COVID has given massive exposure to the healthcare sector since last year. With everything online, data security and privacy is the next most significant concern. This research can be of great support to those working for the security of health care services. This paper provides “Security as a Solution”, which can enhance the security of an otherwise less secure ecosystem. The healthcare use cases discussed in this paper address the most common security issues in the IoT enabled healthcare ecosystem. Future Research: We can enhance this application by including data privacy modules like authentication and authorisation, data encryption and help to abide by the federal privacy laws. In addition, machine learning and artificial intelligence can be extended to other modules of this application. Moreover, this experiment can be easily applicable to many other domains like e-homes, e-offices and many others. For example, e-homes can have devices like kitchen equipment, rooms, dining, cars, bicycles, and smartwatches. Therefore, one can use this application to monitor these devices and detect any suspicious activity.

Aim/Purpose: This paper aims to empirically quantify the financial distress caused by the COVID-19 pandemic on companies listed on Amman Stock Exchange (ASE). The paper also aims to identify the most important predictors of financial distress pre- and mid-pandemic. Background: The COVID-19 pandemic has had a huge toll, not only on human lives but also on many businesses. This provided the impetus to assess the impact of the pandemic on the financial status of Jordanian companies. Methodology: The initial sample comprised 165 companies, which was cleansed and reduced to 84 companies as per data availability. Financial data pertaining to the 84 companies were collected over a two-year period, 2019 and 2020, to empirically quantify the impact of the pandemic on companies in the dataset. Two approaches were employed. The first approach involved using Multiple Discriminant Analysis (MDA) based on Altman’s (1968) model to obtain the Z-score of each company over the investigation period. The second approach involved developing models using Artificial Neural Networks (ANNs) with 15 standard financial ratios to find out the most important variables in predicting financial distress and create an accurate Financial Distress Prediction (FDP) model. Contribution: This research contributes by providing a better understanding of how financial distress predictors perform during dynamic and risky times. The research confirmed that in spite of the negative impact of COVID-19 on the financial health of companies, the main predictors of financial distress remained relatively steadfast. This indicates that standard financial distress predictors can be regarded as being impervious to extraneous financial and/or health calamities. Findings: Results using MDA indicated that more than 63% of companies in the dataset have a lower Z-score in 2020 when compared to 2019. There was also an 8% increase in distressed companies in 2020, and around 6% of companies came to be no longer healthy. As for the models built using ANNs, results show that the most important variable in predicting financial distress is the Return on Capital. The predictive accuracy for the 2019 and 2020 models measured using the area under the Receiver Operating Characteristic (ROC) graph was 87.5% and 97.6%, respectively. Recommendations for Practitioners: Decision makers and top management are encouraged to focus on the identified highly liquid ratios to make thoughtful decisions and initiate preemptive actions to avoid organizational failure. Recommendation for Researchers: This research can be considered a stepping stone to investigating the impact of COVID-19 on the financial status of companies. Researchers are recommended to replicate the methods used in this research across various business sectors to understand the financial dynamics of companies during uncertain times. Impact on Society: Stakeholders in Jordanian-listed companies should concentrate on the list of most important predictors of financial distress as presented in this study. Future Research: Future research may focus on expanding the scope of this study by including other geographical locations to check for the generalisability of the results. Future research may also include post-COVID-19 data to check for changes in results.

Aim/Purpose: Using Artificial Intelligence with Deep Learning (DL) techniques, which mimic the action of the brain, to improve a student’s grammar learning process. Finding the subject of a sentence using DL, and learning, by way of this computer field, to analyze human learning processes and mistakes. In addition, showing Artificial Intelligence learning processes, with and without a general overview of the problem that it is under examination. Applying the idea of the general perspective that the network gets on the sentences and deriving recommendations from this for teaching processes. Background: We looked for common patterns of computer errors and human grammar mistakes. Also deducing the neural network’s learning process, deriving conclusions, and applying concepts from this process to the process of human learning. Methodology: We used DL technologies and research methods. After analysis, we built models from three types of complex neuronal networks – LSTM, Bi-LSTM, and GRU – with sequence-to-sequence architecture. After this, we combined the sequence-to- sequence architecture model with the attention mechanism that gives a general overview of the input that the network receives. Contribution: The cost of computer applications is cheaper than that of manual human effort, and the availability of a computer program is much greater than that of humans to perform the same task. Thus, using computer applications, we can get many desired examples of mistakes without having to pay humans to perform the same task. Understanding the mistakes of the machine can help us to under-stand the human mistakes, because the human brain is the model of the artificial neural network. This way, we can facilitate the student learning process by teaching students not to make mistakes that we have seen made by the artificial neural network. We hope that with the method we have developed, it will be easier for teachers to discover common mistakes in students’ work before starting to teach them. In addition, we show that a “general explanation” of the issue under study can help the teaching and learning process. Findings: We performed the test case on the Hebrew language. From the mistakes we received from the computerized neuronal networks model we built, we were able to classify common human errors. That is, we were able to find a correspondence between machine mistakes and student mistakes. Recommendations for Practitioners: Use an artificial neural network to discover mistakes, and teach students not to make those mistakes. We recommend that before the teacher begins teaching a new topic, he or she gives a general explanation of the problems this topic deals with, and how to solve them. Recommendations for Researchers: To use machines that simulate the learning processes of the human brain, and study if we can thus learn about human learning processes. Impact on Society: When the computer makes the same mistakes as a human would, it is very easy to learn from those mistakes and improve the study process. The fact that ma-chine and humans make similar mistakes is a valuable insight, especially in the field of education, Since we can generate and analyze computer system errors instead of doing a survey of humans (who make mistakes similar to those of the machine); the teaching process becomes cheaper and more efficient. Future Research: We plan to create an automatic grammar-mistakes maker (for instance, by giving the artificial neural network only a tiny data-set to learn from) and ask the students to correct the errors made. In this way, the students will practice on the material in a focused manner. We plan to apply these techniques to other education subfields and, also, to non-educational fields. As far as we know, this is the first study to go in this direction ‒ instead of looking at organisms and building machines, to look at machines and learn about organisms.

The inherent ambiguity in reconstructed images from coherent diffraction imaging (CDI) poses an intrinsic challenge, as images derived from the same dataset under varying initial conditions often display inconsistencies. This study introduces a method that employs the Noise2Noise approach combined with neural networks to effectively mitigate these ambiguities. We applied this methodology to hundreds of ambiguous reconstructed images retrieved from a single diffraction pattern using a conventional retrieval algorithm. Our results demonstrate that ambiguous features in these reconstructions are effectively treated as inter-reconstruction noise and are significantly reduced. The post-Noise2Noise treated images closely approximate the average and singular value decomposition analysis of various reconstructions, providing consistent and reliable reconstructions.

Here, a machine-learning method based on a kinetically informed neural network (NN) is introduced. The proposed method is designed to analyze a time series of difference electron-density maps from a time-resolved X-ray crystallographic experiment. The method is named KINNTREX (kinetics-informed NN for time-resolved X-ray crystallography). To validate KINNTREX, multiple realistic scenarios were simulated with increasing levels of complexity. For the simulations, time-resolved X-ray data were generated that mimic data collected from the photocycle of the photoactive yellow protein. KINNTREX only requires the number of intermediates and approximate relaxation times (both obtained from a singular valued decomposition) and does not require an assumption of a candidate mechanism. It successfully predicts a consistent chemical kinetic mechanism, together with difference electron-density maps of the intermediates that appear during the reaction. These features make KINNTREX attractive for tackling a wide range of biomolecular questions. In addition, the versatility of KINNTREX can inspire more NN-based applications to time-resolved data from biological macromolecules obtained by other methods.

Spectroscopic data, particularly diffraction data, are essential for materials characterization due to their comprehensive crystallographic information. The current crystallographic phase identification, however, is very time consuming. To address this challenge, we have developed a real-time crystallographic phase identifier based on a convolutional self-attention neural network (CPICANN). Trained on 692 190 simulated powder X-ray diffraction (XRD) patterns from 23 073 distinct inorganic crystallographic information files, CPICANN demonstrates superior phase-identification power. Single-phase identification on simulated XRD patterns yields 98.5 and 87.5% accuracies with and without elemental information, respectively, outperforming JADE software (68.2 and 38.7%, respectively). Bi-phase identification on simulated XRD patterns achieves 84.2 and 51.5% accuracies, respectively. In experimental settings, CPICANN achieves an 80% identification accuracy, surpassing JADE software (61%). Integration of CPICANN into XRD refinement software will significantly advance the cutting-edge technology in XRD materials characterization.

Mineral identification and quantification are key to the understanding and, hence, the capacity to predict material properties. The method of choice for mineral quantification is powder X-ray diffraction (XRD), generally using a Rietveld refinement approach. However, a successful Rietveld refinement requires preliminary identification of the phases that make up the sample. This is generally carried out manually, and this task becomes extremely long or virtually impossible in the case of very large datasets such as those from synchrotron X-ray diffraction computed tomography. To circumvent this issue, this article proposes a novel neural network (NN) method for automating phase identification and quantification. An XRD pattern calculation code was used to generate large datasets of synthetic data that are used to train the NN. This approach offers significant advantages, including the ability to construct databases with a substantial number of XRD patterns and the introduction of extensive variability into these patterns. To enhance the performance of the NN, a specifically designed loss function for proportion inference was employed during the training process, offering improved efficiency and stability compared with traditional functions. The NN, trained exclusively with synthetic data, proved its ability to identify and quantify mineral phases on synthetic and real XRD patterns. Trained NN errors were equal to 0.5% for phase quantification on the synthetic test set, and 6% on the experimental data, in a system containing four phases of contrasting crystal structures (calcite, gibbsite, dolomite and hematite). The proposed method is freely available on GitHub and allows for major advances since it can be applied to any dataset, regardless of the mineral phases present.

In cellulo crystallization is a rare event in nature. Recent advances that have made use of heterologous overexpression can promote the intracellular formation of protein crystals, but new tools are required to detect and characterize these targets in the complex cell environment. The present work makes use of Mask R-CNN, a convolutional neural network (CNN)-based instance segmentation method, for the identification of either single or multi-shaped crystals growing in living insect cells, using conventional bright field images. The algorithm can be rapidly adapted to recognize different targets, with the aim of extracting relevant information to support a semi-automated screening pipeline, in order to aid the development of the intracellular protein crystallization approach.

X-ray reflectometry (XRR) is a powerful tool for probing the structural characteristics of nanoscale films and layered structures, which is an important field of nanotechnology and is often used in semiconductor and optics manufacturing. This study introduces a novel approach for conducting quantitative high-resolution millisecond monochromatic XRR measurements. This is an order of magnitude faster than in previously published work. Quick XRR (qXRR) enables real time and in situ monitoring of nanoscale processes such as thin film formation during spin coating. A record qXRR acquisition time of 1.4 ms is demonstrated for a static gold thin film on a silicon sample. As a second example of this novel approach, dynamic in situ measurements are performed during PMMA spin coating onto silicon wafers and fast fitting of XRR curves using machine learning is demonstrated. This investigation primarily focuses on the evolution of film structure and surface morphology, resolving for the first time with qXRR the initial film thinning via mass transport and also shedding light on later thinning via solvent evaporation. This innovative millisecond qXRR technique is of significance for in situ studies of thin film deposition. It addresses the challenge of following intrinsically fast processes, such as thin film growth of high deposition rate or spin coating. Beyond thin film growth processes, millisecond XRR has implications for resolving fast structural changes such as photostriction or diffusion processes.

Due to the ambiguity related to the lack of phase information, determining the physical parameters of multilayer thin films from measured neutron and X-ray reflectivity curves is, on a fundamental level, an underdetermined inverse problem. This ambiguity poses limitations on standard neural networks, constraining the range and number of considered parameters in previous machine learning solutions. To overcome this challenge, a novel training procedure has been designed which incorporates dynamic prior boundaries for each physical parameter as additional inputs to the neural network. In this manner, the neural network can be trained simultaneously on all well-posed subintervals of a larger parameter space in which the inverse problem is underdetermined. During inference, users can flexibly input their own prior knowledge about the physical system to constrain the neural network prediction to distinct target subintervals in the parameter space. The effectiveness of the method is demonstrated in various scenarios, including multilayer structures with a box model parameterization and a physics-inspired special parameterization of the scattering length density profile for a multilayer structure. In contrast to previous methods, this approach scales favourably when increasing the complexity of the inverse problem, working properly even for a five-layer multilayer model and a periodic multilayer model with up to 17 open parameters.

Recent developments in synchrotron radiation facilities have increased the amount of data generated during acquisitions considerably, requiring fast and efficient data processing techniques. Here, the application of dense neural networks (DNNs) to data treatment of X-ray diffraction computed tomography (XRD-CT) experiments is presented. Processing involves mapping the phases in a tomographic slice by predicting the phase fraction in each individual pixel. DNNs were trained on sets of calculated XRD patterns generated using a Python algorithm developed in-house. An initial Rietveld refinement of the tomographic slice sum pattern provides additional information (peak widths and integrated intensities for each phase) to improve the generation of simulated patterns and make them closer to real data. A grid search was used to optimize the network architecture and demonstrated that a single fully connected dense layer was sufficient to accurately determine phase proportions. This DNN was used on the XRD-CT acquisition of a mock-up and a historical sample of highly heterogeneous multi-layered decoration of a late medieval statue, called `applied brocade'. The phase maps predicted by the DNN were in good agreement with other methods, such as non-negative matrix factorization and serial Rietveld refinements performed with TOPAS, and outperformed them in terms of speed and efficiency. The method was evaluated by regenerating experimental patterns from predictions and using the R-weighted profile as the agreement factor. This assessment allowed us to confirm the accuracy of the results.

As a proof of concept, a recurrent neural network (RNN) model was developed using electronic medical record (EMR) data capable of continuously assessing a child's risk of mortality throughout an ICU stay as a proxy measure of illness severity.

Technical framework for deep neural network model partition and collaborative execution

Framework for audio structuralizing based on deep neural networks

Canary Speech's Paired Neural Networks Technology is a form of voice biomarker technology that identifies subtle shifts in an individual's voice by analyzing it against previous samples from the same person.

T. De Ryck and S. Mishra
Math. Comp. 93 (), 2643-2677.
Abstract, references and article information

Umut Güçlü
Jul 8, 2015; 35:10005-10014
BehavioralSystemsCognitive

Words offer a unique opportunity to separate the processing mechanisms of object subcomponents from those of the whole object, because the phonological or semantic information provided by the word subcomponents (i.e., sublexical information) can conflict with that provided by the whole word (i.e., lexical information). Previous studies have revealed some of the specific brain regions and temporal information involved in sublexical information processing. However, a comprehensive spatiotemporal neural network for sublexical processing remains to be fully elucidated due to the low temporal or spatial resolutions of previous neuroimaging studies. In this study, we recorded stereoelectroencephalography signals with high spatial and temporal resolutions from a large sample of 39 epilepsy patients (both sexes) during a Chinese character oral reading task. We explored the activated brain regions and their connectivity related to three sublexical effects: phonological regularity (whether the whole character's pronunciation aligns with its phonetic radical), phonological consistency (whether characters with the same phonetic radical share the same pronunciation), and semantic transparency (whether the whole character's meaning aligns with its semantic radical). The results revealed that sublexical effects existed in the inferior frontal gyrus, precentral and postcentral gyri, temporal lobe, and middle occipital gyrus. Additionally, connectivity from the middle occipital gyrus to the postcentral gyrus and from postcentral gyrus to the fusiform gyrus was associated with the sublexical effects. These findings provide valuable insights into the spatiotemporal dynamics of sublexical processing and object recognition in the brain.

Hi all, I've trained a device model with the approach of an artificial neural network, and it shows well fit. 

May I know how to transfer the trained model to Verilog-A, so that, we can use this model to do circuit simulation?

And I've searched for some lectures that provide the Verilog-A code in the appendix, but I'm freshman in the field of Verilog-A, 

could anyone tell me each statement? such as

real hlayer-w[0:(NI*NNHL)-1   

Convolutional neural networks are useful for classifying grazing-incidence small-angle X-ray scattering patterns. They are also useful for classifying real experimental data.

X-ray reflectivity (XRR) is a powerful and popular scattering technique that can give valuable insight into the growth behavior of thin films. This study shows how a simple artificial neural network model can be used to determine the thickness, roughness and density of thin films of different organic semiconductors [diindenoperylene, copper(II) phthalocyanine and α-sexithiophene] on silica from their XRR data with millisecond computation time and with minimal user input or a priori knowledge. For a large experimental data set of 372 XRR curves, it is shown that a simple fully connected model can provide good results with a mean absolute percentage error of 8–18% when compared with the results obtained by a genetic least mean squares fit using the classical Parratt formalism. Furthermore, current drawbacks and prospects for improvement are discussed.

Tags: Jukebox, music, neural network, OpenAI

In this work, we investigate multi-task learning as a way of pre-training models for classification tasks in digital pathology. It is motivated by the fact that many small and medium-size datasets have been released by the community over the years whereas there is no large scale dataset similar to ImageNet in the domain. We first assemble and transform many digital pathology datasets into a pool of 22 classification tasks and almost 900k images. Then, we propose a simple architecture and training scheme for creating a transferable model and a robust evaluation and selection protocol in order to evaluate our method. Depending on the target task, we show that our models used as feature extractors either improve significantly over ImageNet pre-trained models or provide comparable performance. Fine-tuning improves performance over feature extraction and is able to recover the lack of specificity of ImageNet features, as both pre-training sources yield comparable performance.

A standard approach to evaluating language models analyzes how models assign probabilities to valid versus invalid syntactic constructions (i.e. is a grammatical sentence more probable than an ungrammatical sentence). Our work uses ambiguous relative clause attachment to extend such evaluations to cases of multiple simultaneous valid interpretations, where stark grammaticality differences are absent. We compare model performance in English and Spanish to show that non-linguistic biases in RNN LMs advantageously overlap with syntactic structure in English but not Spanish. Thus, English models may appear to acquire human-like syntactic preferences, while models trained on Spanish fail to acquire comparable human-like preferences. We conclude by relating these results to broader concerns about the relationship between comprehension (i.e. typical language model use cases) and production (which generates the training data for language models), suggesting that necessary linguistic biases are not present in the training signal at all.

Deep learning-based models, such as convolutional neural networks, have advanced various segments of computer vision. However, this technology is rarely applied to seismic shot gather noise localization problem. This letter presents an investigation on the effectiveness of a multi-scale feature-fusion-based network for seismic shot-gather noise localization. Herein, we describe the following: (1) the construction of a real-world dataset of seismic noise localization based on 6,500 seismograms; (2) a multi-scale feature-fusion-based detector that uses the MobileNet combined with the Feature Pyramid Net as the backbone; and (3) the Single Shot multi-box detector for box classification/regression. Additionally, we propose the use of the Focal Loss function that improves the detector's prediction accuracy. The proposed detector achieves an AP@0.5 of 78.67\% in our empirical evaluation.

We present a new system, EEV, for verifying binarized neural networks (BNNs). We formulate BNN verification as a Boolean satisfiability problem (SAT) with reified cardinality constraints of the form $y = (x_1 + cdots + x_n le b)$, where $x_i$ and $y$ are Boolean variables possibly with negation and $b$ is an integer constant. We also identify two properties, specifically balanced weight sparsity and lower cardinality bounds, that reduce the verification complexity of BNNs. EEV contains both a SAT solver enhanced to handle reified cardinality constraints natively and novel training strategies designed to reduce verification complexity by delivering networks with improved sparsity properties and cardinality bounds. We demonstrate the effectiveness of EEV by presenting the first exact verification results for $ell_{infty}$-bounded adversarial robustness of nontrivial convolutional BNNs on the MNIST and CIFAR10 datasets. Our results also show that, depending on the dataset and network architecture, our techniques verify BNNs between a factor of ten to ten thousand times faster than the best previous exact verification techniques for either binarized or real-valued networks.

In recent years, deep neural networks have been applied to obtain high performance of prediction, classification, and pattern recognition. However, the weights in these deep neural networks are difficult to be explained. Although a linear regression method can provide explainable results, the method is not suitable in the case of input interaction. Therefore, an explainable deep neural network (ExpDNN) with explainable layers is proposed to obtain explainable results in the case of input interaction. Three cases were given to evaluate the proposed ExpDNN, and the results showed that the absolute value of weight in an explainable layer can be used to explain the weight of corresponding input for feature extraction.

We empirically evaluate an undervolting technique, i.e., underscaling the circuit supply voltage below the nominal level, to improve the power-efficiency of Convolutional Neural Network (CNN) accelerators mapped to Field Programmable Gate Arrays (FPGAs). Undervolting below a safe voltage level can lead to timing faults due to excessive circuit latency increase. We evaluate the reliability-power trade-off for such accelerators. Specifically, we experimentally study the reduced-voltage operation of multiple components of real FPGAs, characterize the corresponding reliability behavior of CNN accelerators, propose techniques to minimize the drawbacks of reduced-voltage operation, and combine undervolting with architectural CNN optimization techniques, i.e., quantization and pruning. We investigate the effect of environmental temperature on the reliability-power trade-off of such accelerators. We perform experiments on three identical samples of modern Xilinx ZCU102 FPGA platforms with five state-of-the-art image classification CNN benchmarks. This approach allows us to study the effects of our undervolting technique for both software and hardware variability. We achieve more than 3X power-efficiency (GOPs/W) gain via undervolting. 2.6X of this gain is the result of eliminating the voltage guardband region, i.e., the safe voltage region below the nominal level that is set by FPGA vendor to ensure correct functionality in worst-case environmental and circuit conditions. 43% of the power-efficiency gain is due to further undervolting below the guardband, which comes at the cost of accuracy loss in the CNN accelerator. We evaluate an effective frequency underscaling technique that prevents this accuracy loss, and find that it reduces the power-efficiency gain from 43% to 25%.

Recent works imply that the channel pruning can be regarded as searching optimal sub-structure from unpruned networks.

However, existing works based on this observation require training and evaluating a large number of structures, which limits their application.

In this paper, we propose a novel differentiable method for channel pruning, named Differentiable Markov Channel Pruning (DMCP), to efficiently search the optimal sub-structure.

Our method is differentiable and can be directly optimized by gradient descent with respect to standard task loss and budget regularization (e.g. FLOPs constraint).

In DMCP, we model the channel pruning as a Markov process, in which each state represents for retaining the corresponding channel during pruning, and transitions between states denote the pruning process.

In the end, our method is able to implicitly select the proper number of channels in each layer by the Markov process with optimized transitions. To validate the effectiveness of our method, we perform extensive experiments on Imagenet with ResNet and MobilenetV2.

Results show our method can achieve consistent improvement than state-of-the-art pruning methods in various FLOPs settings. The code is available at https://github.com/zx55/dmcp

Spiking neural networks (SNNs) are considered as a potential candidate to overcome current challenges such as the high-power consumption encountered by artificial neural networks (ANNs), however there is still a gap between them with respect to the recognition accuracy on practical tasks. A conversion strategy was thus introduced recently to bridge this gap by mapping a trained ANN to an SNN. However, it is still unclear that to what extent this obtained SNN can benefit both the accuracy advantage from ANN and high efficiency from the spike-based paradigm of computation. In this paper, we propose two new conversion methods, namely TerMapping and AugMapping. The TerMapping is a straightforward extension of a typical threshold-balancing method with a double-threshold scheme, while the AugMapping additionally incorporates a new scheme of augmented spike that employs a spike coefficient to carry the number of typical all-or-nothing spikes occurring at a time step. We examine the performance of our methods based on MNIST, Fashion-MNIST and CIFAR10 datasets. The results show that the proposed double-threshold scheme can effectively improve accuracies of the converted SNNs. More importantly, the proposed AugMapping is more advantageous for constructing accurate, fast and efficient deep SNNs as compared to other state-of-the-art approaches. Our study therefore provides new approaches for further integration of advanced techniques in ANNs to improve the performance of SNNs, which could be of great merit to applied developments with spike-based neuromorphic computing.

Convolutional neural networks (CNN) have shown promising results for end-to-end speech recognition, albeit still behind other state-of-the-art methods in performance. In this paper, we study how to bridge this gap and go beyond with a novel CNN-RNN-transducer architecture, which we call ContextNet. ContextNet features a fully convolutional encoder that incorporates global context information into convolution layers by adding squeeze-and-excitation modules. In addition, we propose a simple scaling method that scales the widths of ContextNet that achieves good trade-off between computation and accuracy. We demonstrate that on the widely used LibriSpeech benchmark, ContextNet achieves a word error rate (WER) of 2.1\%/4.6\% without external language model (LM), 1.9\%/4.1\% with LM and 2.9\%/7.0\% with only 10M parameters on the clean/noisy LibriSpeech test sets. This compares to the previous best published system of 2.0\%/4.6\% with LM and 3.9\%/11.3\% with 20M parameters. The superiority of the proposed ContextNet model is also verified on a much larger internal dataset.

Neural networks have opened up many new opportunities to utilize remotely sensed images in meteorology. Common applications include image classification, e.g., to determine whether an image contains a tropical cyclone, and image translation, e.g., to emulate radar imagery for satellites that only have passive channels. However, there are yet many open questions regarding the use of neural networks in meteorology, such as best practices for evaluation, tuning and interpretation. This article highlights several strategies and practical considerations for neural network development that have not yet received much attention in the meteorological community, such as the concept of effective receptive fields, underutilized meteorological performance measures, and methods for NN interpretation, such as synthetic experiments and layer-wise relevance propagation. We also consider the process of neural network interpretation as a whole, recognizing it as an iterative scientist-driven discovery process, and breaking it down into individual steps that researchers can take. Finally, while most work on neural network interpretation in meteorology has so far focused on networks for image classification tasks, we expand the focus to also include networks for image translation.

Making changes to a program to optimize its performance is an unscalable task that relies entirely upon human intuition and experience. In addition, companies operating at large scale are at a stage where no single individual understands the code controlling its systems, and for this reason, making changes to improve performance can become intractably difficult. In this paper, a learning system is introduced that provides AI assistance for finding recommended changes to a program. Specifically, it is shown how the evaluative feedback, delayed-reward performance programming domain can be effectively formulated via the Monte Carlo tree search (MCTS) framework. It is then shown that established methods from computational games for using learning to expedite tree-search computation can be adapted to speed up computing recommended program alterations. Estimates of expected utility from MCTS trees built for previous problems are used to learn a sampling policy that remains effective across new problems, thus demonstrating transferability of optimization knowledge. This formulation is applied to the Apache Spark distributed computing environment, and a preliminary result is observed that the time required to build a search tree for finding recommendations is reduced by up to a factor of 10x.

Recurrent Neural Networks have been widely used to process sequence data, but have long been criticized for their biological implausibility and training difficulties related to vanishing and exploding gradients. This paper presents a novel algorithm for training recurrent networks, target propagation through time (TPTT), that outperforms standard backpropagation through time (BPTT) on four out of the five problems used for testing. The proposed algorithm is initially tested and compared to BPTT on four synthetic time lag tasks, and its performance is also measured using the sequential MNIST data set. In addition, as TPTT uses target propagation, it allows for discrete nonlinearities and could potentially mitigate the credit assignment problem in more complex recurrent architectures.

The success of Deep Learning and its potential use in many safety-critical applicationshas motivated research on formal verification of Neural Network (NN) models. In thiscontext, verification involves proving or disproving that an NN model satisfies certaininput-output properties. Despite the reputation of learned NN models as black boxes,and the theoretical hardness of proving useful properties about them, researchers havebeen successful in verifying some classes of models by exploiting their piecewise linearstructure and taking insights from formal methods such as Satisifiability Modulo Theory.However, these methods are still far from scaling to realistic neural networks. To facilitateprogress on this crucial area, we exploit the Mixed Integer Linear Programming (MIP) formulation of verification to propose a family of algorithms based on Branch-and-Bound (BaB). We show that our family contains previous verification methods as special cases.With the help of the BaB framework, we make three key contributions. Firstly, we identifynew methods that combine the strengths of multiple existing approaches, accomplishingsignificant performance improvements over previous state of the art. Secondly, we introducean effective branching strategy on ReLU non-linearities. This branching strategy allows usto efficiently and successfully deal with high input dimensional problems with convolutionalnetwork architecture, on which previous methods fail frequently. Finally, we proposecomprehensive test data sets and benchmarks which includes a collection of previouslyreleased testcases. We use the data sets to conduct a thorough experimental comparison ofexisting and new algorithms and to provide an inclusive analysis of the factors impactingthe hardness of verification problems.

The transfer learning technique is widely used to learning in one context and applying it to another, i.e. the capacity to apply acquired knowledge and skills to new situations. But is it possible to transfer the learning from a deep neural network to a weaker neural network? Is it possible to improve the performance of a weak neural network using the knowledge acquired by a more powerful neural network? In this work, during the training process of a weak network, we add a loss function that minimizes the distance between the features previously learned from a strong neural network with the features that the weak network must try to learn. To demonstrate the effectiveness and robustness of our approach, we conducted a large number of experiments using three known datasets and demonstrated that a weak neural network can increase its performance if its learning process is driven by a more powerful neural network.